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[57] ABSTRACT 

A central processing unit (CPU) is coupled to a computer 
display for displaying graphic and other information. The 
CPU is further coupled to a pointer control device such as a 
pea. stylus or mouse that permits the user to selectively 
position a pointer and activate an action handle on the 
display associated with a data object Activation of the 
action handle signals the CPU of selections associated with 
the data object Tapping or clicking on the action handle 
causes display of a context menu containing at least one 
command that may be invoked with respect to the data 
object. Dragging the action handle indicates movement or 
dragging of the action handle and the corresponding data 
object for an operation associated with dragging such as 
drag-and-drop. The use of action handles in association with 
data objects eliminates ambiguity between actions intended 
to create digital ink with respect to a pen-based computer 
system and actions intended to invoke a command or other 
interaction with the computer system. 

30 Claims, 12 Drawing Sheets 
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METHODS AND APPARATUS FOR device, the 4 'where you are" is represented by the indicator 

INTERACTING WITH DATA OBJECTS itself, and. optionally, by a cursor. 

USING ACTION HANDLES Because there is no keyboard or mouse, pen-based com- 
Tcrtnjrn n puters have tighter restrictions on what a user can and cannot 
TECHNICAL FIELD 5 do. Common user interfaces such as those is found in the 
The present invention relates generally to apparatus and Microsoft WINDOWS™ operating system and other GUI- 
methods for displaying and manipulating information oa a based computer systems typically rely on a keyboard and a 
computer display screen, and more particularly relates to a pointing device with buttons such as a mouse or trackball for 
computer controlled display system far displaying action user interaction. The user points to a location on the screen 
handles or icons associated with varying types of data 10 where an action is desired, and invokes a command by 
objects on a display screen, and for allowing user interaction pressing a button on the pointing device or entering a 
with selected data objects by way of the action handles or keyboard command. For example, to insert a space between 
icons. words it is generally assumed the user can simply select a 

location for insertion of the space by using the mouse to 

BACKGROUND OF THE INVENTION 15 position a cursor at the point of insertion and hit the space 

Present day computer systems often employ "object- bar Consequently many present-day application software 

oriented" displays with icons, pictures, text, pop-up or Programs do not offer an "insert space command for text 

drop-down menus, dialog boxes, and other graphical items editing. 

or objects in what is known as a "Graphical JJser Some known computer application programs add corn- 
Interface" or "GUT. In such systems, various combinations 20 mands to existing drop-down menu bars. However, this 
of graphical items or objects are utilized to convey infor- method is not suitable for handheld pen-based applications 
motion to users via a cathode ray tube (CRT) monitor or since most menu bars are impracticably large on devices 
display. Users generally interact with the computer system with a small screen. 

by activating graphical items or objects on the display. The M Therefore, systems that are required to work with a pen or 

graphical items or objects are often activated by pointing at stylus must find alternative methods to accomplish such 

them with a mouse, pen. or other pointing device, and actions. 

clicking or tapping a button. I„ pen-based computers, it is difficult for the computer to 
Object oriented display systems often utilize small graph!- distinguish actions of a user intended as digital ink from 
cal items or objects called "icons**, which symbolically M actions intended to invoke operations of or provide corn- 
indicate a type of operation the computer system will mands to the computer system. It is desirable that a user 
execute if the icon is selected by a user interface device such interface for a pen-based computer system be able to inter- 
as a mouse, pen. or keyboard. These icons are frequently pret movement of the stylus in a consistent unambiguous 
disposed within a window on a display. Icons are often used fashion, so that users can enter data or invoke commands 
to delete or copy information, move information from one 33 quickly and without frustration. It is also important that the 
window to another, cause operations to the data in the user interface be intuitive and easy to remember, so that the 
window, and the like. An example of a computer graphical user can readily determine how to tell the computer that he 
user interface that uses icons is the Microsoft WINDOWS™ or she intends to invoke a command, or make digital ink (e.g. 
operating system manufactured by the assignee of the enter text via handwriting) 

present invention, Microsoft Corporation. ^ o nc example of a known pen-based computer system is 

New generations of computer systems that use a pen or the Newton™ MessagePad™ manufactured by Apple 

stylus as a user input device are being introduced. Some of Computer. Inc.. Cupertino. Calif. This device is a small, 

the new pen-based computers are small, handheld devices hand-held device with a liquid crystal display (LCD) screen, 

with a display screen but no keyboard. Such devices are do keyboards or buttons (other than an on-off switch), and 

sometimes called "Personal Digital Assistants**. "PDAs", or 45 a stylus that is used to interact with the system. The user 

"portable pen pads". With such devices, a user interacts with manipulates the stylus to enter handwriting, tap on regions 

a display screen like a sheet of paper, using an ink-less pen of the screen, or draw pictures. The Apple Newton attempts 

or stylus to draw pictures, to enter text via handwriting or to interpret patterns of markings by the stylus on the screen 

printing, and to interact with the computer system. The or "gestures" as commands. A gesture is a symbol or pattern 

computer interprets the motion of the pen or stylus relative 50 of markings that the user can draw with a stylus (or finger) 

to the display screen, and often displays lines or markings on that is recognized by the system as a command, rather than 

the screen that track the movement of the pen. Markings on as text or a picture. 

the screen displayed as a result of movementof the stylus are For example, in the Apple Newton a squiggly gesture 

often called digital ink*. made on top of a word like this: means "delete this** 

The pen or stylus in a pen-based cornputing system may 55 ^ (if mtcrprctcd properly) causes the deletion of the object 

be considered a direct pointmg device , that is. a pointing OT tcxt which mc squiggly ^ was ^ wtL However, 

device consisting of a display and an indicator such as a ^ ^ - mtapKt ^ ^ mc 

finger, stylus, or other object. The pointing-device actions of ^ and &c rcsult would ^ a ^^11^ WO rd VordW". 
•pointing and "tappingT — which are analogous to "click- 

ing" with a mouse — are achieved by moving the indicator on 60 Likewise, a caret gesture like this: toqjclose means to 

or near the display screen and touching the display screen, "insert space**. Moreover, the user must remember that a 

respectively. With a mouse or trackball type pointing device, squiggly gesture means "delete**, and that a caret gesture 

there is an indirection. The user moves some physical device means "insert**, since there is no readily apparent means for 

(positioned somewhere away from the screen) in order to telling the user what the gestures are if forgotten, 

move an artificial pointer (Le. a cursor) on the display 65 Stated in other words, gestures are "hidden" from the user 

screen. Without the cursor, there is no way for the user to and can be ambiguous. That is, there is no way for a user to 

"see where you are** on the screen. With a direct pointing determine what particular gesture to draw when a particular 
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command is desired, except to be told or remembered. Once 
learned, however, gestures can be easily forgotten, if not 
repeatedly used Another disadvantage is mat of recognition. 
Gestures can be and frequently are misrecognized. The 
results of a rnisrecognized gesture range from simple annoy- 
ance to unintended loss of data. 

Even with traditional keyboard and/or mouse-type graphi- 
cal user interfaces, there is sometimes ambiguity as to the 
function of an icon or the manner for invoking a command 
for a data object displayed on screen. For example, in a 
mouse-driven GUI. if users wish to delete a word from 
within a sentence, they must remember that they must (1) 
first select the word to be deleted by pointing and dragging 
the cursor (thereby forming a "selected data object") and (2) 
then Invoke the appropriate "delete" command by either (a) 
remembering the keyboard equivalent command, (b) finding 
the command from a drop-down or pop-up menu which may 
be remotely located on the screen relative to the selected 
data object, or (c) activate a disassociated "delete" icon 
somewhere on the screen, assuming they can find it 

While all of such methods (keyboard command, menu 
command, icon command) are presently employed with 
success in modern application computer programs, it would 
advance the art of GUI design if the object itself upon which 
an action is desired could provide the user with the options 
to invoke actions in a concise, consistent and uniform 
methodology. Such an advance would not force users to 
remember keyboard commands, locate commands in menu 
located away from the object, or locate appropriate action 
icons located remotely from the object In yet other words, 
it would be desirable that selected data objects themselves 
provide a context for prompting and receiving user interac- 
tion. 

In order to reduce ambiguity to some degree, some 
application programs for the Apple Newton Message Pad 
system employ fixed-position icons that display pop-up 
menus of selectable items when tapped. (A "tap" is a form 
of gesture wherein the user lightly strikes a spot on the 
display screen with the stylus.) These fixed-position icons 
are typically a small, diamond shaped region (e.g. ♦ ) whose 
very shape signifies that it is a "hot spot". It has proven easy 
for users to remember that the diamond icon means 4 lap here 
for display of selectable options." 

However, such pop-up menus are associated with a series 
of selectable items, and are not context-sensitive. There is no 
predictable one-to-one connection between an object and an 
icon. In other words, selected data objects themselves do not 
provide a context for prompting and receiving user interac- 
tion. Moreover, such pop-up menu icons do not completely 
duplicate gesture functionality, and cannot be moved or 
repositioned. 

It is expected that the use of pen-based computing sys- 
tems will increase in the foreseeable future because of the 
flexibility afforded by such systems to people who either 
cannot or do not wish to type using a keyboard. Moreover, 
pen-based computers offer the advantage of greater port- 
ability since no keyboards or pointing devices are required 
other than a simple stylus. It is foreseen that even future 
desktop computer systems may utilize flat panel display 
screens that serve both as an upright display when a user 
wishes to use a keyboard and/or mouse for interaction, but 
may also be removed from a stand, placed in a lap or on a 
desktop, and employed like "digital paper" with a stylus, as 
the user enters "digital ink". Therefore, there is a need for 
computer operating systems mat offer increased command 
accessibility and reduced ambiguity, especially for pen- 



4 

based computer systems, but for that matter with any types 
of computer systems that rely upon undiscoverable or easily 
forgettable actions that are associated with particular icons. 

5 SUMMARY OF THE INVENTION 

The present invention overcomes the foregoing described 
difficulties with present computer user interface design by 
providing action handles or icons in association with data 
objects, typically displayed in close proximity to a related 

10 data object displayed on the screen. The action handles 
provide a method for performing actions on text and other 
data objects with a pen or other pointing device, and also a 
method for displaying a context menu for the text or other 
objects. A context menu, for purposes of the present 

15 application, is a menu of commands that apply to only a 
particular object within a particular context and that pops up 
on screen over, or in proximity to the particular object 

Briefly described, the present invention comprises com- 
puter systems and methods that comprise displaying a small 

20 icon or "action handle" next to or in close proximity to a data 
object, such as a selected text item, or a calendar item, or a 
list of other data items, or at an insertion point within a range 
of text The action handles receive event messages indicative 
of a tapping or dragging action with a stylus or other 

25 pointing device such as a mouse. In response to a tapping 
action, preferably a context menu displaying commands 
available for the object or text is displayed. In further 
response to selection of a command in the context menu, the 
system carries out the appropriate selected command. In 

30 response to a dragging action, preferably the selected object 
is moved to a new location. 

For example, an action handle next to an address in an 
address book can be dragged to copy or move the address to 

33 another application, for example, to an electronic mail 
("e-mail") application to send mail to a person. Or. the action 
handle can be tapped to cause display of the context menu 
that offers commands that are appropriate in a given context 
such as "delete-, "prinf, or "edit", that act on the address. 

40 As another example, the action handle associated with a 
selected text data object might be used to move the text 
around by dragging, and might contain commands in its 
context menu such as "delete", "make upper case", "make 
bold face". Similarly, an action handle associated with an 

43 insertion point (which may include a draggable cursor) can 
be dragged to select text which then yields an action handle 
associated with the selected text The selected text action 
handle itself, like the draggable cursor action handle, can be 
tapped to reveal commands such as "insert space" or "start 

x new line". 

Preferably, the action handle will contain graphical infor- 
mation to assist in identifying the object in question. For 
example, the action handle for an address item in an address 
book might look like a business card. 

55 More particularly described, the present invention relates 
to methods and systems for displaying and controlling 
actions relating to a data object displayed on a display in a 
computer controlled display system having a display 
coupled to a central processing unit (CPU). The methods and 

60 systems include the steps of generating a graphic image 
associated with the data object on the display, and generating 
a graphic image of an action handle proximate to the data 
object An activatable region on the display associated with 
the action handle graphic image is provided. A user positions 

65 a pointer on the display using a pointing control device 
operatively associated with the CPU, which thereby pro- 
vides a signal to the CPU upon correspondence of the 
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pointer with the activatabie region. In response to a signal drag-and-drop and selection are very difficult fox a computer 

indicative of a first type of interaction between the pointer to distinguish from inking and writing operations. Action 

and the activatahlc region, the method involves displaying at handles solve this problem. 

least one selectable command relating to the data object In It will thus be appreciated that action handles work with 

response to signal indicative of a second type of interaction 5 any type of pointing device, and provide a mechanism 

between the pointer and the activatabie region, the method whereby a user can readily determine that an action is 

involves moving the graphic image of the data object on the possible. This solves the problem of creating context menus 

display in computer application programs that rely upon a "right 

In the preferred embodiment, the first type of interaction f"* b " t,on ( for wU * a ^-P5 ns hav ' n ° 

\ . , . Jr T ,~ for activation of a context menu. With an action handle, 

is a tap while the second type of interaction 1S a drag. These to a user can click 

types of interactions are particularly suitable for pen-based cr tap on to get a context menu. 

computer systems. The action handle upon its creation can indicate a visible 

The preferred step of displaying at least one selectable encapsulation of an object being manipulated. In other 

command comprises displaying a context menu on the words, the icon can represent the object for actions to be 

display including a plurality of selectable commands. In the 15 taken , with the object Action handles further permit a 

context menu, there is provided an activatabie second region single-tap or click method to cause display of the context 

on the display associated with the selectable command. In menu. This contrasts with certain double-click or other 

response to interaction between the second region and the multiple tap or click methodologies which are difficult for 

pointer, a computer command corresponding to the select- users to remember. 

able command is executed, ft-eferabry. again, the interaction 20 The action handle provides a single focus of the user's , 

between the pointer and the second region comprises a tap. attention for all actions on an object, both dragging and 

Still more particularly described, the step of moving the command execution. Again, this helps eliminate ambiguity 

graphic image of the data object on the display comprises and promotes consistency in the user interface across dif- 

moving the graphic image of the action handle on the display ferent applications and through the operating system, 

during a dragging operation from a first position to a second Action handles, as contemplated by the present invention, 

position on the display. In response to a release command are automatically generated and exposed in the appropriate 

provided at the second position on the display (typically by context. If any particular action is available for an particular 

withdrawing the pen or stylus from the display screen), a object, the action handle is displayed. Thus, the user need 

predetermined operation is carried out e.g. the graphic not remember any gestures, and can always find out what 

image of the data object is moved to a new position on the action to take by either dragging the action handle or tapping 

display associated with the second position, or if the object the action handle. Even common insertion-point icons or 

is "dropped" onto a clip box region, the data object is copied. cursors can automatically appear in edit controls. 

In the invention, the data object may be a data item Accordingly, it is an object of the present invention to 

associated with a predetermined application program. For 35 provide an improved graphical user interface for computer 

example, the data item could be a selected string of text systems. 

associated with a text processing application program, or a it is another object of the present invention to provide an 

calendar entry associated with a calendar application pro- improved graphical user interface particularly suitable for 

gram. pen-based computer systems. 

It will be appreciated mat the invention has utility for both ^ it is another object of the present invention to provide an 

pen-based and mouse-based (e.g. desktop) computer sys- improved graphic user interface for pen-based computer 

terns. Advantageously, action handles are very discoverable systems that reduces ambiguity in the possible actions that 

and cannot be forgotten since they are always displayed in can be taken with respect to an object displayed on a display 

association with the data object. To find out what command screen. 

arc available in the context of a particular object, the user 45 u is another object of the present invention to provide an 

can simply tap on the action handle to display the context unambiguous method for selecting text and other objects in 

menu, and men selects an appropriate command from the list a pen-based computer system graphic user interface, 

of choices in the context menu. Execution of a command in ^ is another object of the present invention to provide for 

the context menu operates like other menus. an improved user interface for pen-based computers that 

Action handles also have no ambiguity about them. The 50 allows a computer system to distinguish actions of a user 

actions available at any given time are always displayed by intended as digital ink from actions intended to invoke 

tapping the action handle to reveal the context menu, and if operations of or provide commands to the computer system, 

the item is movable, the action handle icon, as it is dragged n is another object of the present invention to provide a 

with the stylus or pointing device on the screen, can change pen-based user interface for a computer system mat is able 

its graphical properties to indicate that movement is permit- 55 to interpret movement of a pen or stylus in a consistent 

ted. If movement is not permitted, the action handle simply unambiguous fashion, so that users can enter data or invoke 

never moves or pops back to the original location, indicating commands quickly and without frustration, 

non-movability to a particular location. This would be handy ^ other objects, features, and advantages of the 

in applications wherein a particular data item cannot be vnm% invention may be more clearly understood and 

discarded, as when a user attempts to drag the non- 60 appreciated from a review of the following detailed descrip- 

discardable data object to the trash can or a delete icon, and tion ^ the disclosed embodiments and by reference to the 

the icon pops back to its location original. appended drawings and claims. 

Action handles therefore eliminate the need for gestures. 

Action handles always provide a visible place where the BRIEF DESCRIPTION OF THE DRAWING 

stylus always acts as a pointing device rather than as a pen 65 FIGURES 

(that is. no digital ink is created when touching an action FIG. 1 illustrates the general architecture of a handheld 

handle). In text entry areas, mouse operations such as pen-based computer that is constructed in accordance with , 
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and utilizes the teachings of (he present invention to provide 
action handles in association with data objects. 

FIG. 2 illustrates an exemplary display screen for the 
computer of FIG. 1. that illustrates various relationships of 
action handles and associated data objects. 

FIG. 3 illustrates exemplary static action handles utilized 
in the preferred embodiment of the invention to indicate 
different types of data objects or items. 

FIG. 4 illustrates exemplary dynamic (in dragging 
motion) and status indicating action handles or "drag icons" 
utilized in the preferred embodiment of the invention to 
indicate status of a drag or a drag-and-drop operation. 

FIG. 5 illustrates a general drag-and-drop operation uti- 
lizing an action handle. 

FIG. 6 illustrates a drag-and-drop operation to a clip box. 
for copying a data object associated with an action handle. 

FIGS. 7A and 7B illustrates pop-up or context menus that 
are generated in the preferred embodiment of the present 
invention when a user taps on an action handle associated 
with a predetermined application and a predetermined data 
item, respectively. 

FIGS. SASC illustrates a drag-and-drop operation 
applied to an action handle associated with a range of 
selected text in an exemplary text-based application 
program, in accordance with the invention. 

FIG. 9 illustrates an action handle that is "pinned" to a 
border of a window in accordance with the invention, 
indicating that the associated data object has been scrolled 
off-screen. 

FIG. 10 illustrates an action handle associated with a 
small range of selected text of a size approximating that of 
the action handle itself, displayed beneath the selected text 
in accordance with the invention. 

FIGS. 11A-11G illustrates an action handle in association 
with a flashing cursor data object for purposes of text 
selection and manipulation, in accordance with the inven- 
tion. 

FIG. 12 is a table illustrating a general class of objects (a 
rectangle) with associated instance variables, messages, and 
methods, in connection with object-oriented programming. 

FIG. 13 is a table illustrating an exemplary Icon class of 
objects with associated instance variables, messages, and 
methods, with corresponding subclasses of Program Icon 
and Action Handle Icon, in accordance with the invention. 

FIG. 14 is a table illustrating a hierarchy of object classes 
utilized in constructing a computer program that implements 
the preferred embodiment of the present invention. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

Referring now to the drawings, in which like numerals 
illustrate like elements throughout the several views. FIG. 1 
illustrates a pen-based computer system 10 for generating 
graphic images on a graphic display 18 and receiving user 
interactions by way of a pen or stylus 12. Although the 
present invention is described in conjunction with a pen- 
based computer, it will be appreciated that the present 
invention may be utilized in conventional (desk top or 
portable) computer systems that use a keyboard or cursor 
keys to position a cursor on the display screen, as well as a 
mouse, trackball, or other pointing device that moves a 
cursor on the display and often includes a switchbutton to 
indicate a command. 

In the following detailed description, numerous details are 
provided such as computer display system elements, object 
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definitions, display formats, sample data. etc. in order to 
provide an understanding of the invention. However, those 
skilled in the art will understand that the present invention 
may be practiced without the specific details. Well-known 

5 circuits, programming methodologies, and structures are 
utilized in the present invention but are not described in 
detail in order not to obscure the present invention. 

Certain of the detailed descriptions which follow are 
presented in terms of exemplary display images, algorithms. 

10 and symbolic representations of operations of data bits 
within the computer* s memory. As will be known to those 
skilled in the programming arts, and particularly those 
skilled in the object-oriented programming methodologies 
(about which more is explained later), these algorithmic 

15 descriptions, class descriptions, messages, notification 
descriptions, and graphic exemplary displays are the means 
used by those skilled in the art of computer programming 
and computer construction to convey teachings and discov- 
eries to others skilled in the art 

20 For purposes of this discussion, an "algorithm" is gener- 
ally a sequence of computer-executed steps leading to a 
desired result These steps require physical manipulations of 
physical quantities. Usually, though not necessarily, these 
quantities take the form of electrical, magnetic, or optical 

25 signals that are capable of being stored, transferred, 
combined, compared, or otherwise manipulated. It is con- 
ventional for those skilled in the art to refer to these signals 
as bits, values, elements, symbols, characters, images, terms, 
numbers, or the like. It should be kept in mind, however, that 

30 these and similar terms should be associated with appropri- 
ate physical quantities inside the computer and that these are 
merely convenient labels applied to these physical quantities 
that exist within the computer. 

35 It should also be understood that manipulations within the 
computer are often referred to in terms such as adding, 
comparing, moving, etc. which are often associated with 
mental operations performed by a human operator. It must 
be understood that no involvement of a human operator is 

40 necessary or even desirable in the present invention, since 
the operations described herein are machine operations 
performed in conjunction with a human operator or user that 
interacts with the computer. The machines used for perform- 
ing the operation of the present invention, as will be ' 

43 understood, include general purpose digital computers or 
other similar computing devices. 

Furthermore, it should be kept in mind that there is a 
distinction between the methods, steps, or operations carried 
out by a computer, and the method of computation itself. The 

50 present invention does not involve a method of computation. 
The present invention rather relates to methods, steps, or 
operations for a computer and processing electrical or other 
physical signals to generate desired physical signals and 
display results and interactions. As illustrated in FIG. 1, the 

55 present invention also relates to apparatu s 10 for performing , 
these operations. This apparatus may be especially con- 
structed for the described purposes, e.g.. a touch-sensitive 
pen-based computer system, or it may comprise a general 
purpose computer that is activated or operated by computer 

go programs stored in the computer. 

Furthermore, it should be understood that the programs, 
algorithms, objects, etc described herein are not related or 
limited to any particular computer or apparatus. Rather, 
various types of general purpose machines may be used with 

65 programs constructed in accordance with the teachings 
herein. Similarly, it may prove advantageous to construct 
specialized apparatus to perform the method steps described 
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herein by way of dedicated computer systems with hard- responds by repositioning the text within the available 
wired logic or programs stored in nonvolatile memory such window. In the WINDOWS™ operating system, when a 

as read only memory. user resiles a window, the operating system sends a message 

to the application program indicating the new window size. 

OBJECT ORIENTED PROGRAMMING 5 The program then adjusts the contents of its window to 

Those skilled in the art will understand and appreciate that refletf the new size. 

£ be* mode for implementing the present invention is via Jf* TX^U^lX^ S3E 

object^nented programming languages and techniques. function (a procedure or a method) wim^JHhe program!^ie 

^ JC ^ C r PJSF !TT g tC ^ mqUCS " amsu *b 10 parameters tf this message describe the particular^ function 

utilized by those skilled in the art of computer prograrnming 10 ^ CTCCUted by mc com™^. 

because they provide a powerful way to view the interaction m idea ofpassing a to a program is tantamount 

of a user with a computer system. Since people interact with to mMn a a function within the program. Such a 

objects, it is logical to attempt to write computer program so functioD call can be made to the operating system, or to an 

that the * objects with which a person interacts on a com- application program. Those skilled in the art will understand 

puter screen have counterparts "inside the computer", so to 15 such mechanisms are how a program opens a disk file, 

s P ca ^- for example, or scrolls text from an area that is "off screen" 

For example, one particular "object** that people often into view within the window. An operating system may 

work with on computer screens is called a "window". A make to a program by passing it messages, in a similar 

"window" is a rectangular area on a display screen that is manner that the application program makes a call to the 

used to display information or receive interactions. Indeed. 20 operating system 

the display screen itself is a window of sorts into the Every window object that a program creates has an 
computer. A window receives user input from a keyboard, associated window procedure (also called a 'method"). This 
mouse, or pen. and displays graphical output on its surface. window procedure or method is a function that could be 
A computer program is responsible for generating the either in the program itself » in a linked library of appro- 
window, and responding to user interactions. A window 25 pn^tr routines that can be executed upon a call. The WIN- 
often contains the computer program's title bar across the DOWS™ operating system sends a message to a window by 
top of the window, menus, sizing borders, and perhaps other calling a window procedure. The window procedure does 
objects. some processing based on the message passed to it and then 

The surface of a window may contain additional smaller M returns control to the operating system, possibly with a 

windows called "child windows**, which are subspecies or "notification** or other response. (A "notification" is merely 

subclasses of windows. Child windows may take the form of a message passed back to a calling program, and usually 

push buttons, radio buttons, check boxes, text entry fields. indicates status information or the results of an operation.) 

list boxes, and scroll bars, and the like. These objects do not A window displayed on a display screen is based on what 

truly exist as physical objects, but when displayed on the 35 is known as a "window class** of objects. The window class 

computer screen may be pressed or clicked or scrolled like of objects identifies a particular procedure that processes 

they have physical existence. A user sees a window as an messages passed to that window. The use of a window dass ' 

object on the screen, and interacts directly with the object by allows multiple windows to be based on the same window 

pushing "a button**, or scrolling a scroll bar, or tapping a class, and thus use the same window procedure. For 

handle. ^ example, all control buttons in programs that utilize the 

A skilled object-oriented programmer's perspective is WINDOWS™ operating system are based on the same 

analogous to a user's perspective. A window generated by a window class. A window class is associated with a window 

computer program receives user input in the form of mes- procedures that processes messages to all button windows, 

sages to the window. The messages are generated by an In object-oriented programming, an "object** is a corabi- 

event handling routine that is responsive to user input such 45 nation of computer program code and data. A window is a 

as tapping, clicking or dragging a cursor with a stylus or type of object The computer program code carries out tasks 

mouse. associated with the object The data is the formation 

The following is a brief example of an object-oriented retained by the window procedure and information retained 

programming methodology for resizing a window utilized in by the WINDOWS™ operating system, for each window 

the Microsoft WINDOWS™ graphical user interface oper- 50 411(1 window class that exists. 

ating system. Consider a word processor application com- A window procedure processes messages to the window, 

puter program that displays text in a window. The window Very often these messages inform a window of a user input 

may include a "size box**, which is a control located at the from a keyboard, mouse, or pen. This is how a child window 

lower right-hand corner of the window. If selected and that is a button "knows" that it is being jessed", or an icon 

dragged, the size box causes the size of the window on the 55 1 "knows** that it is being dragged. Other messages tell a 

display screen to change to larger or smaller. If the pro- window when It is being resized or when the surface of a 

gram's window is resized, the word processing program window needs to be repainted. 

application program will reformat and move the text on the When a program begins execution in such an operating 
display screen to fit within the window when the window is system as the WINDOWS™ operating system, the operating 
resized. 60 system creates a * 'mes sage queue** far the program. This 
The computer's operating system program generally message queue stores messages to all the various windows 
handles the details of resizing the window, and the applied- a program may create. The program includes a portion of 
tion program running under control of the operating system program code called the "message loop** to retrieve the 
program responds to the system function of resizing the messages from the queue and dispatch mem to the appro- 
window. The word processing program "knows** when its 65 priate window procedure. Other messages may be sent 
window is resized because a message is passed to it indi- directly to a window procedure without being placed in the 
eating a resizing operation, and the application program message queue. 
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Preferred Handheld Computer System 

With the preceding background in mind, turn next to FIG. 
1 for a discussion of the preferred embodiment of the 
handheld pen-based computer If constructed to carry out 
the methods of the present invention. The pen-based com- 5 
puter 10 comprises various components, which are illus- 
trated graphically for purposes of illustration. First, a rect- 
angular liquid crystal display (LCD) 18 occupies the front 
surface of a housing 19. and provides a surface upon which 
graphic images are displayed. Preferably, the LCD display 10 
18 is touch or electromagneticaUy sensitive. A stylus 12 that 
is held by a user is used to write, draw, or gesture upon the 
surface of the display 18 to allow user interaction with the 
computer. 

The computer 10 is preferably constructed around a 
central processing unit (CPU) 20, a memory 21. and an 
input/output (I/O) circuit 22. all of which are physically 
contained within the housing 19 but are shown externally in 
FIG. 1 for ease of understanding. It will be understood that ^ 
the I/O circuit 22. CPU 20. and memory 21 are those 
typically found in most general purpose computers. Indeed, 
computer 10 is intended to be representative of a broad 
category of data processing devices. Thus, although there is 
no keyboard or other pointing device shown in FIG. 1. it will ^ 
be understood that such types of computer systems are also 
suitable for use with the present invention. 

The I/O circuit 22 is used to communicate information in 
appropriately structured form to and from other portions of 
the computer 10. For example, the I/O circuit 22 drives an ^ 
optical port 25 which is employed to communicate infor- 
mation optically to similar optical devices. The I/O circuit 
further connects to a serial port 27. which allows electrical 
connection of peripheral devices such as cornmunication 
lines to a desktop personal computer system, modem, or 33 
other device. The I/O circuit further connects to a PCMCIA 
slot 29 which allows usage of a PCMCIA memory card or 
other peripheral PCMCIA device such as shown at 31. 

The display 18 is shown coupled to the I/O circuit 22 and 
is used to display images, data. etc. generated by the CPU 20 ^ 
in accordance with the present invention. The display, as will 
be known to those skilled in the art is a rectangular array of 
picture elements or "pixels", beginning at an origin (x=0. 
y=0). and extending across the width of the display screen 
to the rightmost extent (x=W), and downwardly to the 43 
lowermost extent (y=H) of the display screen. 

In accordance with the invention, shown on the display 
screen 18 are two particular graphic display elements or 
items that in general characterize the present invention. 
Firstly, there is shown an action handle or icon 40 positioned 50 
adjacent to a text object 42. Preferably, both of these graphic 
display elements are objects in the object-oriented computer 
programming sense. 

In accordance with the present invention, an action handle 
or icon 40 is displayed in proximity to (such as adjacent or 35 
atop) an associated data item or object such as the text 42; 
user interaction with the action handle 40, and therefore with 
the associated data object 42, is permitted in various ways 
described herein, without requiring gestures. Data items that 
utilize the present invention will always include an associ- 60 
ated action handle in proximity on the display screen. 

In the example shown in FIG. 1. exemplary text object 42 
happens to be a string of text that is displayed in a rectan- 
gular region on the display screen 18 defined by the dotted 
box. (It wiD be understood that the dotted box b not 63 
displayed on the display screen of the computer, but merely 
signifies an exemplary data object — the text contained 
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within the rectangular boundaries shown — for purposes of 
ease of understanding. Moreover, the invention contem- 
plates various types of data objects, not merely text objects, 
e.g. a cursor object a drawing object, an icon object, etc.) 

The action handle 40 typically is relatively small com- 
pared to the data item, although size is not critical and no 
limitation as to size is intended. The size of the action handle 
40 should preferably be such that a user can readily locate 
the action handle associated with any data object on the 
display screen. Indeed, the presence of an action handle 
indicates the presence of a selectable data item or object and 
the ability for the user to interact with the data object 

The exemplary action handle 40 shown in FIG. 1 com- 
prises an activatable object, region, or "hot spot" on the 
display screen. This hot spot is bounded on the display 
screen by a predetermined region along the x-axis between 
x=HSl and x=HS2. and along the y-axis between y=HS3 
and y=HS4, If the user places the tip 13 stylus 12 within the 
hot spot region defined by these coordinates, the touch- 
sensitive circuitry of the display screen 18 will detect the 
presence of the stylus within the hot spot and generate an 
appropriate message. In a pen-based computer system, two 
particular actions of the stylus relative to the screen are 
contemplated — a tap and a drag. A tap occurs when a user 
quickly places the tip of the stylus within the hot spot and 
lifts it up. tapping the surface lightly on the display screen. 
A drag occurs when the user places the tip of the stylus into 
the hot spot and rather than Lifting the stylus, leaves the tip 
of the stylus in contact with the surface of the display screen 
and moves the stylus along a path on surface of the screen. 
In accordance with the present invention, the computer will 
generate appropriate graphical symbols on the display 
screen that will follow the tip of the stylus around on the 
display screen as if the user were "dragging" the action 
handle. 

The foregoing actions of tapping and dragging allow the 
computer 10 to differentiate between actions that are 
intended by a user as an interaction with the data object or 
with the computer system, versus actions that are intended 
to constitute the entry of data such as handwriting or 
drawing on the display screen, both of which generate 
"digital ink" such as shown at 14. As has been previously 
described, "digital ink** 14 is an illuminated region of pixels 
on the display screen generated by the computer system in 
response to the movement of the stylus tip along the touch- 
sensitive surface of the screen. 

Before turning to another subject, it will be understood 
that in accordance with the invention* an object such as the 
exemplary text object 42. and its associated action handle 
such as the action handle 40. are selectably repositionable on 
the screen. In other words, should the object be moved to 
another location on the screen, as by invocation of a 
command, a scroll of the window, a dragging operation, etc.. 
the action handle maintains its relative position in accor- 
dance with the associated data object No matter where a 
particular data object appears, if it has an action handle 
associated therewith, the action handle will be displayed in 
appropriated proximate juxtaposition. Those skilled in the 
art will understand that programming for the computer to 
implement the present invention will need to take into 
account the relocatabUity of a data object, and provide for 
corresponding relocation of the associated action handle. 

FIG. 2 illustrates various different relationships of action 
handles 40 with associated data objects 42. and the preferred 
relationship contemplated as the best mode by the present 
inventors. Several different embodiments of action handles 
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and associated data objects are shown. The upper region of For a selected range of text on a single line, the action 

the display screen 18 includes a list of data items or objects, handle is preferably placed immediately above or 

which in this case happens to be a list of names and below the text, wherever there is more visible room, 

telephone numbers in an Addresses window. An exemplary centered horizontally with respect to the selected text 

name in the list, 4t Bill Clinton", is shown highlighted as a 3 so as to avoid obscuring the text 

selected data object 42* with an association handle For a sclccted string of text mat spans more than one line. 

^ S * e ' " " ^ such as shown at 42c in FIG. 2. the action handle is 

Tiolodex card-shaped icon positioned immediately to the _ . . . . . ^ , , . , Ct . 

left of the assc^teddala hern 42*. preferably placedm the center of aboundmg box of the 

Inthe central region of the display screen 18 in FIG. 2. 10 selected text. A bounding ^xtf me smallest rectangle 

maeisshownasclectedgrc^ JtowaMt^tou^toL 

Eric" and bishop. Byron". Both names are highlighted, Wlth thc exceptions noted above, if any of the foregoing 

indicating that the data items are selected. The associated for action handlc placement cause the action 

action handle 40b comprises an icon bearing the res em- handle to be off screen or outside the visible bounds of 

blance of a stack of multiple sheets of paper. In the preferred the window in which the object or text Is displayed, the 

embodiment, this particular icon indicates selection of mill- 15 action handle is preferably moved vertically and/or 

tiple items. Thus, it will be understood that a data object can horizontally until it is just inside a bounding rectangle 

comprise plural data items considered as a group; an action °f me window or screen on the display. If only one 

handle such as 406 associated with such a group operates coordinate (x or y) exceeds the bounds of the window, 

upon and with the group of 4?T? items. then only that one coordinate needs to be adjusted. 

In the lower portion of the display screen 18 in FIG. 2. 20 ^ final «k is most ^ t0 ** utaizcd whcn 

there is shown a data object 42c comprising string of window constitutes a scrollable pane, which can occur if a 

highlighted text which in this case consists of the words « selected by the user, and the pane is scrolled 

"WinPad provides drag icons by which the user can manipu- so ^ thc selected text and its associated action handle are 

late selections and objects on the WinPad screen". Thc scrolled off screen. Preferably, then, the action handle is 

associated action handle 40c consists of a small rectangular 25 displayed in the manner described even though the associ- 

box containing the lowercase letter "a" and is positioned atcd object " «* visible to the user. This is described 

atop the selected text in greater detail below. 

In the upper right hand of FIG. X there is shown a data *? GS - 3 4 Nustrate exemplary action handles^ in 

object 42* which in this case comprises a tlip box" for „ Referred static and dynamic forms, respectively m FIG 3. 

carrying out clipboard operations such as copying. An 30 is shown a series of different exemplary action handles 

associated action handle 404 consists of a small rectangular *** too "^ 4ftfc whose i«™ indicate the nature of the 

box positioned inside the clip box. As contemplated by (he "soaatcd data object, which are listed. Smtfariy. FIG. 4 

invention, if the user wishes to copy certain information to dustrates the display of dynamic stams^dicating icons 

a different page or application, the user can use the clip box „ that arc displayed as an action handle is dragged from one 

as a temporary holding area for the information. For 33 [°^ on to ?noAer on the display screen. Note tot each 

example, the user can copy a name (such as the name "Bill <^«nt ac^on handle has a different grapmc symbol or icon 

OintotO to the dip box by dragging an action handle 40c associated therewith The graphic nature of the icon mdi- 

associatedwimmenametomeclipbox.Thedata(mename) cates Aenatinx of me associated i^ 

w01 be copied into a memory buffer, and available f or ^ 111 Ha 3 - handlc lC0n 40 ° comprising a 

'Wing" into another program or into another object Thc 40 lowercase letter "a" is utilized to indicate a text data item, 

presence of an action handle such as 4<W shown in the clip T hc i( ? n 4W \ is *? fm ™ y * Ust t0 m *^J£f* 

box indicates that a data item has previously been "copied" cn ^°pe icon 40c indicates a message. A Yoiodex 

into the clip box. Preferably, thc action handle can be card icon 40J is used to indicate an address. A miniature 

dragged out of the dip box to perform a paste operation. M ™ n J° c » us ^!° incUcatc » llcm ; * 

7^ ni A , . — *• ♦.• 3 pencil icon 4Qfis used to indicate a note data item. A stacked 

It will therefore be appreciated that the present invention v' c J . „ . . ^ . . . . ... . 

i * • • * - r u • sheets of paper icon 40* is used to indicate that multiple 

contemplates provision of various types of action handles, in . . . . ^ * . . . _ . jl, 

a:k . JZ~ ~f /j«t9 «ku^. t« objects have been selected. An insertion point/caret icon 4»h 

association with various different types of data objects. In . £ . . . . ... r . . 

r , ... t r * ■ is preferably used in association with an insertion point 

the preferred embodiment of the present invention, action ,„ . » ... # ^ # . . . 

. 57 MA r . . . A . ^ , -* «■ i (flashing bar) to indicate actions that can be taken with 

handles 40 arc preferably positioned on the duplay 18 a, „ ^ to sdect texU or tepping t0 

tollows: , , . . . — . . 

. , , , ■ _ . , . ^_ invoke predetermined commands). 

For a sxn^e oata object such a smg^e name ^ i^he Hst m ^ ^ ^ na 3 generally are dis- 

the action handle is preferably placed to the left of the Really in the present invention in association with 

object (Note-^erabty. this icon is ^owedto scroU £ ^ ^ of Q type. The action handle 

off screen if the associated object is scrolled off screen.) 55 remain£ displayed as long as the data object is visible on the 

For a group of objects, such as a selected series of screen in mc form shown< 
addresses 426 in the address list the action handle is 

placed to the left of the objects, but vertically centered Action Handle Drag and Drop Operation 

through the vertical extent of the selected group of It will be recalled mat one of the allowed actions is to drag 

objects. (Note — preferably, this icon is allowed to 60 an action handle, for purposes of moving the associated data 

scroll off screen only if all associated objects are item, or copying it, or invoking another action. As will be 

scrolled off screen.) known to those skilled in the art. it is a customary practice 

For an insertion point, which is typically a flashing caret in many computer graphical user interfaces for a program to 

positioned somewhere in a string of text or on a generate and display a different icon from the static icon 

drawing figure, the action handle is placed just beneath 65 when an object is dragged with a pointing device such as 

the baseline of the insertion point cursor or beneath the stylus or mouse. The use of a different icon during morion 

baseline of the text generally provides certain predetermined status information 
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about the action undertaken, for example that the item 
associated with the icon has been selected and capable of 
being moved, or that a time delay is occurring, or that a 
given action is not permitted. For purposes of the present 
invention, an icon displayed during an action such as drag- 
ging is called a "dynamic" or "status indicating" icon* and 
may also be called a "drag icon.** 

Drag icons are preferably utilized in the present invention 
during certain fjredetermined operations, namely, when an 
action handle is dragged or otherwise activated. FIG. 4 
illustrates various dynamic action handles or drag icons 
40a-40g employed in the preferred embodiment. A selected 
one of these drag icons is generated and displayed "under- 
neath** the stylus point or other pointer as the stylus or 
pointer is ( 1) first used to point to and select an action handle 
by placing the stylus tip onto the hot spot of the action 
handle (or placing a pointer onto the hot spot and clicking to 
select the action handle, if a mouse or trackball is used), and 
(2) then moved about on the display screen to "drag* the 
action handle. Generally during a drag operation, the static 
action handle (as shown in FIG. 3) remains displayed in 
position with its associated data item, and the computer CPU 
20 generates one of the exemplary drag icons as shown in 
FIG. 4 until the user 'releases" the drag icon by withdrawing 
the stylus away from the display screen (or by clicking or 
other appropriate action if another type of pointing device is 
used). 

The exemplary drag icons of FIG. 4 provide visual 
feedback to a user about the status of the current operation, 
such as a drag and drop operation. For example, a copy 
pointer drag icon 40a indicates that a copy operation has 
been invoked, that the selected data item has been copied, 
and that a copy of the selected item is being moved. A 
multiple objects pointer drag icon 40b indicates that a 
selected group of objects is being moved from one location 
to another. 

The user can also determine that the data item can not be 
dropped at the current pointer location (e.g. when the user 
attempts to drop the selected data item on or in an incom- 
patible or impermissible location) when the drag icon 
appears as a circle with a slash through it as shown at 40c. 

Other types of move and copy type drag icons are shown 
at 40J-40g. In the preferred embodiment, a user can deter- 
mine whether a copying operation or moving operation has 
been activated by the presence or absence of a plus sign (+) 
associated with the drag icon. For example, the "copy" type 
drag icons 40tL 40e include a plus sign (+)• while similar but 
<4 move M type drag icons 40/, 40g lack the plus (+) sign. 

fteferably. the selection of a copy type icon or a move 
type icon is determined at the time an object is dropped, and 
depends upon the drop target. For example, if a drag icon is 
moved onto a storage directory or subdirectory, the plus sign 
(+) may be generated to indicate that a drop of the icon 
would result in the item being copied into that directory or 
subdirectory. 

Refer now to FIG. S for a discussion of the manner in 
which a user of a pen-based computer interacts with an 
action handle for a dragging operation, and how the static 
action handles and drag icons are generated. In the figure, 
the action handle 40 is being dragged with a stylus 12, from 
its initial position to a rest position at 53. In this example, the 
data object is a calendar item 42 indicating "12:00— Project 
status meeting with StacT, and has a corresponding (static) 
action handle comprising a rniniature calendar. The static 
action handle 40 remains in position during the operation, 
displayed in proximity to the data object. As the stylus 12 is 
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moved to drag the action handle, a drag icon 40* is generated 
at successive positions along the trajectory 48 of the stylus, 
beginning immediately after the tip of the stylus leaves the 
area of the static drag icon at 51 until the resting position at 

s 53. which is at the calendar time 5:00 FM. At the resting 
position 53. assume that a location where the selected data 
item may properly be deposited or "dropped**, e.g. at the 
time 5:00 PM, which occurs upon release or withdrawal of 
the stylus (or clicking or tapping, if desired). If the end result 

to of the operation is a proper move operation, upon release the 
static action handle 40 and its associated data item 42 appear 
at the new location, and are deleted from the old. initial 
location. 

It will now be appreciated that, in the preferred 
15 embodiment, a user can use action handles to copy or move 
information anywhere within a supported series of applica- 
tions operating within the computer system. The approach 
utilized in the present invention is considered a "drag-and- 
drop** approach, which is generally known to those skilled in 
20 the art The drag-and-drop approach to data movement and 
manipulation is very useful when a user needs to change or 
duplicate information quickly. In the calendar example of 
FIG. 5, if a user wishes to change a meeting to a different 
time of day, he or she can reschedule it by dragging the 
25 action handle 40 associated with an appointment time within 
a calendar to a new tune, or a new date. During the drag 
operation, an appropriate drag icon is generated and dis- 
played. 

Those skilled in the ait will understand that a drag-and- 

30 drop operation depends upon the particular application and 
context. For example, in a calendar application program, 
information that is dragged within a page is considered 
moved, whereas in a different application, such as a "to-do** 
list the information is copied rather than moved. Preferably. 

35 therefore, the system is programmed to presume that a move 
is intended if the drag-and-drop operation occurs within a 
given application, but that a copy operation is intended if the 
data item is posted to a clip box to be copied, or moved to 
a different application program to which the data can prop- 

40 erly be copied. 

Refer now to FIG. 6 for a discussion of the use of action 
handles to move a data item from one application to another. 
As previously described, the present invention supports a 

45 copying operation using the action handle. The clip box 45 
provides a region within the display 18 for providing a 
"clipboard" operation. If the user wishes to copy the infor- 
mation associated with an action handle to a different page 
or application, the user can use the clip box as a temporary 

x holding area for the information. 

For example, assume in FIG. 6 that the user wishes copy 
a data item comprising a note 42 "Product Proposal" from a 
Notes application to a Mail application. The user first drags 
the action handle 40 associated with the data item along the 

33 path 58 to the clip box 45, activates the Mall application, and 
then drags from the clip box into the Mail application. A clip 
box action handle 40d appears in the clip box 45 after this 
first drag operation to indicate that data has been copied into 
the cup box. The movement into the Mail application is 

60 accomplished by dragging the action handle 40W out of the 
clip box along a path 59 to a new location. 

In the preferred embodiment, the clip box 45 only holds 
one item of information at a time. If a second item is copied 
to the clip box by dragging and dropping, then the existing 

65 information is deleted. However, information can be copied 
from the clip box multiple times, as the information is not 
deleted until another item is copied on top of it. 
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Thus, to copy or move information between pages or 
applications, the user: 
Drags the action handle of the associated data object to the 
clip box 45. 

If it is desired to copy or move information to another 
application, then switch to the other application by 
using the stylus, pointing device or other command to 
activate the application. 

Activates other controls in the computer system to display 
the dialog, ox page, where the information is to appear. 

Drag the action handle from the clip box 45 to its new 
location. 

If the user is moving information, the user may switch 
back to the originating or first application and delete the 
original information. 

Action Handle Tapping or Other Selection 

As previously described, a particular action contemplated 
in the present invention is that of tapping an action handle 
with a stylus. The preceding diagrams have illustrated 
operations involving dragging an action handle. In the 
preferred embodiment of the present invention, tapping an 
action handle causes display of a "pop- up" menu, also called 
a "context" menu, that display various types of actions that 
may be selected with respect to the particular data object 
associated with the action handle. Those skilled in the art 
will understand that in computer systems that utilize point- 
ing devices such as a mouse or trackball, the same consid- 
erations will apply, except that the operation is that of 
pointing and clicking as opposed to tapping. 

FIG. 7 illustrates two exemplary types of pop up menus 
contemplated in for use in the present invention. FIG. 7A 
illustrates an application-oriented pop-up or context menu 
70a that appears when application action handle (e.g. a 
Notes action handle 40) is tapped. For purposes of the 
present discussion, an "application" action handle is an 
action handle that is positioned in proximity to a data item 
corresponding to an application computer program, such as 
a word processor, a note taking utility, a calendar program. ^ 
an address list manager, etc. In accordance with the 
invention, the application pop-up menu 70a includes, in a 
predetermined hierarchy, a series of selectable actions that 
may be activated by the user. In FIG. 7A. the selectable 
actions include Properties .... Delete. Clear Ink. Send . . . 43 
. and Print . . . Preferably, the pop-up menu appears on the 
display screen in response to a tap of the action handle, 
overlaid upon other information, and remains until (a) the 
user taps somewhere on the screen outside the menu or 
presses the escape key. or (b) one of the selectable activities x 
has been activated by a tap upon the appropriate desired 
action. For example, to initiate a printing operation, the user 
would tap the Print . . . region, which would be highlighted, 
flashed, and a printing operation would commence. 

Other types of pop-up menus are contemplated. In FIG. 55 
7B. a data item pop-up menu 70b is illustrated. A data item 
pop-up menu is displayed when an action handle associated 
with a data item is tapped. In the example shown, there is a 
list of names, including the name "Benson" shown in 
highlighting (the other names are obscured by the pop-up 60 
menu 706. although their action handles are visible). The 
data item pop-up menu also contains, in a predetermined 
hierarchy, a series of selectable actions that may be activated 
by the user. 

The pop-up menu 7W> displays various actions that are 65 
available for this particular data object. For example, in FIG. 
7B, the operations of Edit .... Delete. Send Print . . 



. . and Help ... are available. As in the application-oriented 
action handle, the user may activate the appropriate action 
by tapping the desired command within the pop-up menu. 

The p re f erre d action handle pop-up or context menus 
provide a user with a consistent mechanism to interact with 
objects in computers that employ the teachings of the 
present invention. In order to maintain consistency between 
application, and preferred embodiment, the rules set forth in 
Table I are employed. A context menu is preferably divided 
into five logical sections. However, not all applications or 
objects will necessarily implement all sections: 

TABLE I 



20 



23 



30 



33 



Fust Item 


The preferred first item in a 




context menu b a default action 




that occurs when the user 




double taps oo the action . 




handle. Thus, the user can 




avoid the delay associated with 




popping up the menu and then . 




selecting the first item by 




merely double tapping the 




actkm handle or the object (or 




double clicking with other 




pointing devices). 


Application 


The second group of 


Specific 


commands is where 




application-specific 




information is placed. For 




example, in an e-mail 




application calfed Mail, this 




could include the commands 




Reply, Reply AH, etc. 


System 


This is where all system 




specific functions are placed 




Examples of these include 




Send . . . , Print . . . , Save .... etc. 


Dckte 


This section contains the delete 




commands for the object and 








include Delete Ink. Delete. 


Properties 


This section contains the 




Properties . . . selection which 




allows a user to select 




properties, attributes, or 




preferences. 



The following Table n is an example of selected exem- 
plary commands in a preferred order as employed in the 
present invention. Those skilled in the art will understand 
that each individual application program can utilize only the 
required menu selection items. Preferably, note also that the 
different sections are preferably separated by heavy border 
so as to allow readily distinguishable regions for the user to 
select. 



TABLE II 



First items 


Edit... 




View 


Application Specific 


Undo 




Cut, Copy, Paste 




Reply, Reply All 




Forward 


System 


Send . 




Print . . . 




Save . . . 


Delete 


Clear Ink 




Dckte 


Properties 


Properties . . . 



Action Handle for Variable Data Such as Text 

FIG. 8. consisting of FIGS. 8A-8C illustrates a sequence 
of operations and associated graphic displays that are gen- 
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erated in the preferred embodiment from an interaction with 
an action handle associated with a range 84 of selected text 
in an exemplary text-based application program. In the 
example shown in FIG. 8. the display 18 includes a window 
82 in which is displayed a string of (admittedly nonsensical) 
text: This is a test of the dog that is lazy. This theory that 

I have, that I say which is mine, is mine. This is a test of the 
foxy hue.** 

Of the entire text string shown, only this text is selected: 
'•test of the dog that is lazy. This theory that I have, that I say 
which is mine, is mine. This a test of the foxy", as shown in 
reverse video (white on black). This is an example of a range 
84 of text that has been selected by an operation described 
in conjunction with FIG. 11. The reader is referred to FIG. 

II for a discussion of the manner in which text is selected 
utilizing a stylus or other input device. 

The range of text 84. which is a data item, has associated 
therewith an action handle 40a. in accordance with the 
invention. 

In the example of FIG. 8. assume that the selected range 
of text 84 is to be moved by a drag-and-drop operation to 
another location, such as after the word "hue". The present 
invention supports dragging the action handle 40a. convert- 
ing the action handle to a cursor to indicate an insertion 
point, and release of the action handle at a desired location, 
resulting in movement of the text to a new location 

In FIG. 8 A, the range of text 84 is selected, and the 
appropriate type of static action handle 40 is displayed, in 
this case a text-type static action handle 40a from FIG. 3. 
centered within a bounding box of the text as set forth in 
earlier-described rules. 

In FIG. 8B. the stylus is placed into the hot spot region of 
the action handle 40a and dragged a short distance away 
from the original static action handle, but still within the 
highlighted region of selected text. Here, the computer 
causes generation of a dynamic action handle or drag icon 
40/. such as the drag icon 40/ from FIG. 4, indicating that a 
move operation is being effected. Note that the particular 
action handle 40/ shown in FIG. 8B corresponds to a 
movement-type (but not copied-type) action handle. As 
previously described, the drag icon provides visual feedback 
to the user that a dragging operation is underway. 

In FIG. 8C, once the x-y coordinates of the stylus tip or 
pointing device passing outside of the selected range of text 
84. two things occur— (1) a flashing cursor or insertion point 
86 is displayed underneath the stylus tip to indicate that the 
data item (the selected text) associated with the action 
handle may be "dropped" at the position indicated by the 
cursor 86. and (2) a different type of drag icon 40# is 
displayed. The user indicates that he or she wishes to drop 
the selection at the desired point by lifting the stylus from 
the surface of the display screen 18 (or by clicking, if 
appropriate). 

Note the difference between the drag icon 40/ and 40#. 
The drag icon 40/ has the tip of the arrow inside a small 
rectangular icon with a dog-eared corner, to indicate that 
something is being dragged. On the other hand, the drag icon 
40g has the base of the arrow inside the rectangular, dog- 
eared icon, and the tip of the arrow tenninating on the 
insertion point 86. to indicate that the item will be put into 
the location pointed at by the arrow. 

Those skilled in the art will understand and appreciate that 
the use of varying types of action handles in the present 
invention-static and drag i con-substantially reduces ambi- 
guity in a GUI employing the teachings of the invention. 
More specifically, the second cursor type makes it easier for 
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a user to see exactly where moved or copied text will be 
placed. As described in the background, there can be ambi- 
guity in a pen-based system as to whether a move operation 
or a copy operation is intended. In some contexts, determi- 

5 nation whether a move operation or copy operation is 
intended depends upon the destination at release of the 
dragged action handle. For example, a dragging operation of 
an action handle into something like a trash can may 
implicitly be interpreted as a "delete object" operation. On 

to the other hand, a drag onto a fax machine icon or the clip box 
45 shown in FIG. 1 is a copy operation. 

It will now be understood that there are at least two types 
of drag icons utilized depending upon the destination type. 
If a dropping operation into a text area such as shown in FIG. 

15 8 is intended, a standard cursor such as shown at 40/ is 
utilized. This type of cursor keeps the hot-point (the tip of 
the arrow) so that the user can see the insertion point or 
cursor. In other cases, however, it may be desirable to show 
the actual action handle or a replica thereof being moved at 

20 the hot spot of a drag cursor. Preferably, a drag operation 
cursor will include an arrow so as to provide visual feedback 
to the user that a dragging operation is being carried out 

In FIG. 8B, as the user drags the action handle, the cursor 
changes and transforms into a drag icon 40# with arrow 

25 comprising a large easy-to-see caret following the cursor 86. 
indicating where the text will go. While the hot-spot (the tip 
of the arrow) is still over the selected text or source 84. the 
drag icon shown at 40/ is preferred. 
As shown in FIG. 8C. if the user moves to another portion 

30 of the document, the hot spot (tip of the arrow) culrninates 
on the flashing insertion bar. 

Referring now to FIG. 9. if the user scrolls to another 
portion of the document, as by using a scrollbar (not shown 

35 in the figure), the action handle remains visible. **pinned n to 
the edge of the screen or document closest to the actual 
position of the selection. In FIG. 9. the action handle 40 is 
pinned to the top border 88 of a window to indicate that the 
selected data item is off screen. If the user stops dragging. 

^ preferably, the action handle transforms from a drag icon 
such as 40/ or 40g to a static action handle such as 40a 
(shown in FIG. 9). 

The user can preferably pick up the action handle and 
resume the drag at any time. Preferably, if the drag is 

45 resumed with the action handle "pressed** to a window 
border such as 88 (or a bottom border such as 89) the text 
will autoscrolL that is. the text in the window scrolls 
upwardly or downwardly when the action handle is moved 
to a border of a display window, as a function of velocity of 

5Q the stylus or other pointing device. 

When the action handle is released by removal of the 
stylus from the screen, the selection appears in a new 
location indicated by the cursor and remains selected or 
highlighted. 

55 It will understood that the scenario just described is one 
where the selected text is moved to another location on the 
screen. It should be remembered that the action handle could 
also have been dropped onto an application button, the clip 
box 45, or other icon. If such other icons are responsive to 

60 the action indicated, appropriate action will be indicated and 
activated. 

Turning now to FIG. 10. if the selected range of text 84 
is not big enough to contain the action handle 40. preferably 
it is drawn adjacent to the selection. Note that the action 
65 handle 40 in FIG. 10 does not obscure the selected text 84. 
which consists of a single word "dog" in this example. 
Preferably, the action handle is displayed beneath the text, in 
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accordance with the rules described eariier herein, so that the of text 'two one 1 *. In one embodiment the action handle can 

user is not prevented from drawing gestures upon the disappear and the southeast arrow cursor 93 remain, as 

selection ox upon other text if desired (ox above the shown. Each time the insertion point crosses the midpoint of 

selection, if there is inadequate room below). a cell in which a text letter is centered, the selection highlight 

5 adjusts accordingly to indicate selection. In the preferred 

Selection of Text With Action Handle and Cursor embodiment, this is indicated by display of the characters in 

- „• „. w tn „ n t . ^ c *«t ;™*«ti~. «ir~ reverse video (white on black). In other words, selection 

Turning next to FIG. 11. me preset mvenUoo also ^ ^ tQ mat ea ^ ed with ^ pres^d-hold 

contemplates association of an action handle with a data £ ^ common ^ user interfa^s. Note that 

object such as a cursor that can be conveniently used for me stylus outside of the action handle 40a would 

operations such as selecting text in a word processing type "> haye handwriting entry. 

appuc^on.oxse^g^ ^ na UD when me sclection hflS ^ 

ing or a picture. The problem addressed by this aspect of the cstablishea . ta example selection of the word *two" 95, 

invention relates to distinguishing digital ink from com- ^ user ^ mc pe D ( 0 r clicks with the mouse). Preferably, 

mands to the computer such as selecting text With a then, the cursor changes back to the pen 92. and a static 

pointing device such as a mouse or trackball, it is known that 15 action handle 40b appears centered inside the selection 95. 

a user can select text by simply (1) pressing a button at the Optionally, the insertion point action handle 40a may be 

beginning of the text to indicate the beginning of a selection. retained at the end of the selection, underneath the insertion 

which u sually causes display of a flashing cursor or insertion point 90. 

point, (2) dragging the curses- across the text and (3) It will be recalled from earlier discussion that if the 

releasing the button to indicate the end of the selection. 20 selection takes up more than one line, the text action handle 

However, this press/drag/release action is ambiguous with a 40b should preferably be centered in the selection. 

stylus pointing device, where it cannot be distinguished Alternatively, if a selection begins near the end of one line 

from handwriting. Therefore, this type of operation is not and ends shortly after the beginning of another line, so that 

preferred for use with a pen, because the action of dragging the selection highlighting on the two lines do not overlap. 

with the pen is preferably reserved for drawing digital ink to 25 then the action handle may be centered in whichever part of 

make letters and/or gestures. th c selection that is longest Similaity. if the selection is so 

» .V. _ . , . . large that part of it is not visible in the displayed window, the 

In the present invention, this problem is solved by asso- . r ,, . _ . , . . *. ' . _ . 

mil uit pi^m xu 3 r . . A . , * action handle is preferably centered inside the visible por- 

aating an action handle with an insertion point in text. This * <.»i*„*L« 

n r j_ .I * ._». \La i _* ■ t> 1100 of the selection. 

allows creauon of a draggable stretchable selection. For n will also be recalled that if the user scrolls a selection 

reposes of this discussion, an^erdon point is the place 30 b M actioD t0 a window border, the action 

where characters typed from a keyboard are placed, typically handle^ooils with the selection until the action handle 

denoted by a blinking caret or vertical bar. An insertion point thc cdgc of mc wmd0 w. If the user continues to 

can also be thought of as a selection containing zero char- SC roll the selection out of sigit the action handle "sticks" or 

acters * is pinned at the edge of the window. This feature allows the 

Optionally, the present invention contemplates generation user to scroll to a distant drag/drop destination and then drag 

and display of the action handle only upon an initial action the still-visible action handle to carry out the drag/drop 

such as a tap by the user at a particular location in the text. operation. 

This particular option requires an additional step — the initial As shown in FIG. HE. in order to move the text the pen 

tap to create the insertion point and associated action handle. (or cursor) is positioned over the text action handle 40b and 

R-eferably. however, an action handle is displayed in asso- 40 the cursor changes to the appropriate dynamic drag icon. e.g. 

riation with a flashing insertion point continuously, so that 40c* 

the user will readily be able to find the insertion point As shown in FIG. 11F. the user presses the pen down on 

(because of the size of the action handle) and will not have the action handle 40c and begins to drag to a new location, 

to remember the tap required to cause creation of the action as 40a*. If desired, the static action handle can disappear, 

handle 45 but preferably is retained, while suitable drag icon tracks the 

t *7Ti~ ha 4 v ~c *u • u ^ u ^ movement of the stylus. Optionally, a thin gray line or 

In FIG. llA.meref^,mereis shown an action handfc insertion cursor may be employed to track thc drag icon, as 

40a displayed ui association wfch acaret : 9C .which ismerely showQ m nG . 11F at 96. Preferably, if the user lifts up the 

f^J 3 ^ 0 ' % f W *^ G ' J 10 ^ " P" while the gray line is still within the highlighted 

indicated. The action handle 40a may be considered a V, %.j ¥mwt . m „ , 

M , . . . _ ^ . ^ . / ^ ... w selection region, the text is not moved, 

selection drag button that is centered beneath the blinking . . . , ^ u 

insertion rxrinlor cursor 90. As shown in FIG. UG when the desired drop location has 

luscruou pumi i* «m W *v. been reached, thc user lifts the pen up and the selected text 

Note also the presence of a pen icon 92. which represents ^ (scJcctcd ^ ^yxt) appears at the new location. The 

an independently displayed icon in systems that are not ^ |§ to me MW location formcrly rcpre . 

pen-based, or can represent die action of a stylus brought 55 ^ ^ moQ of mc flashi mt % 

into contact with region on the display screen defining thc p^^y, ^ ^ TtmahiS selected, with the static icon 40 

action handle 40a. or action handle centered inside. The user can then easily 

In FIG. 11B. assume that the pen (or other pointing movc mc selection again if necessary or can invoke the 

cursor) 92 is positioned ova the action handle 40a and in context menu by tapping on thc action handle to act upon the 

contact with the display screen. Preferably, the pen cursor 60 selection with appropriate commands, as described, 

changes to an arrow 93 (in this case a "southeast" arrow.) T^e skilled in the art will now understand the operation 

Preferably, a southeast arrow cursor is employed rather than of ^ ^ invention to implement action handles in 

the northwest cursor that is often used in mouse-based conjunction with associated data objects, 
systems, because a northwest cursor is too easily obscured 

by the pen in a user's hand. $5 Object Oriented Progranmung of Action Handles 

In FIG. Che pen is pressed down and the user begins Those skilled in the art will understand that object- 

to drag the action handle 40a to the right, towards the words oriented programming languages such as C++ are preferred 
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for preparing computer programs in accordance with the 
present invention, since such programming approaches 
facilitate the paradigm of objects consisting of action 
handles and associated data items. Before discussing spe- 
cific objects, messages, procedures, and notifications 
employed in the construction of the preferred embodiment 
of the present invention, we will first provide a brief 
overview of object-oriented programming, so that the reader 
will understand that those skilled in the art will be enabled 
to make and use the present invention with the descriptions 
provided herein. 

Hie basic difference between procedural and object- 
oriented programming is in the way that the data and action 
upon the data is treated. In procedural programming, data 
and action are two separate things. A programmer defines 
data structures, and then routines are written to operate on 
the data structures. For each data structure that is employed 
a separate set of routines is required 

In object-oriented programming, actions and data are 
more closely related. When a programmer defines the data — 
which constitutes objects — he or she also defines the actions 
that can be taken upon such data. Instead of writing a set of 
routines that do something to a separate set of data* the 
programmer creates a set of objects that interact with each 
other by passing messages back and forth. 

In "object" is an entity that contains some data and an 
associated set of actions that operate on the data. The data 
can be extracted from the object and sent to another object 
for example when the data is to be stored in a file. To make 
an object perform one of its actions, the object is sent a 
message. For example, in the present invention the program- 
mer will create an object that represents the action handle, 
and similarly will create objects that represent data items 
such as an address, a selected range of text, a insertion point, 
or the Like. Trie data associated with the object such as the 
action handle will contain locations of the boundaries of the 
icons and the definition of the hot spots, a bit map of the 
graphic appearance of the action handles, the present coor- 
dinates of the action handles, etc.. as well as indication that 
various actions can be taken upon appropriate activation. 
For example, to cause an action handle to pop up a context 
menu, an action handle object is sent a *tap M message. 

For the time being, the manner in which objects are sent 
messages is beyond the scope of the present discussion. 
Such operations are typically within the purview of the 
operating system or of the compiler mat is employed to 
generate program code. 

Contrast the foregoing with the manner in which an action 
handle is programmed with procedural programming. First, 
the programmer must define a data record that represents the 
boundaries of the icon relative to the display screen, and 
must translate these local boundaries to the global coordi- 
nate system associated with the display screen. The pro- 
gramme? will then have to write different routines for the 
different actions that could be performed when a user 
interacts with the action handle, namely, one to cause a 
pop-up menu to be generated upon a tap. one to cause 
generation of an "inside selection" drag icon as the action 
handle is dragged within the range of selected text, and a 
different routine to cause display of an "out of selection" 
drag icon as the action handle is dragged outside of the 
selected text. These routines would have to be rewritten for 
each different type of action handle and the associated 
actions that could be carried out with respect to that par- 
ticular type of action handle. 

A particular advantage of object-oriented r^gramming 
over procedural programming is that a programmer can keep 
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the routines that operate on a data structure together with the 
data structure that they are operating on. This is called 
"encapsulation". 
A more significant advantage in object-oriented program- 
5 ming is that objects can inherit data and behavior from other, 
similar objects. Every object belongs to a "class", which 
defines the implementation of a particular kind of object A 
class describes the object's data and the messages to which 
it responds. 

10 Classes are somewhat related to record declarations in 
conventional programming languages. A class may be 
thought of as a template for creating objects. The class 
describes an object's data and the messages to which it 
responds. A particular object is called an instance of a class. 

15 The programmer defines data for the class the same way that 
fields of a record are drfined With classes and objects, 
though, fields are called "instance variables". Each instance 
of a class, that is each object, has its own instance variables, 
just as each variable of a record type has the same fields. 

20 When a message is sent to an object the object invokes 
a routine that implements that message. These routines are 
called "methods**. A class definition includes method imple- 
mentations. Method implementations are sequences of com- 
puter program steps that are carried out in response to a 

25 particular message. 

It is important to keep in mind that a message and a 
method are not the same thing. A message is sect to an 
object However, an object responds to a message by car- 

30 rying out a method 

Turn next to FIG. 12 for an illustration of these object- 
oriented concepts. This figure illustrates an exemplary class 
definition 100 for an object type. "Rectangle". Assume for 
purposes of this discussion that a Rectangle object is a 

33 computer program that is operative to draw a rectangle on 
the display screen in response to being sent a "Draw" 
message. For this exemplary Rectangle class, there are 
instance variables consisting of the Upperleft Comer. 
Upperright Corner, Lowerleft Comer, and Lowerright Cor- 

4Q ner. There might also be an instance variable indicative of 
the thickness of the line to be drawn between the corners to 
define the rectangle, Unewidth. There might also be an 
instance variable. Fillpattern. indicative of a fill pattern to be 
filled or drawn inside the boundaries of a rectangle. 

45 Three exemplary messages are shown for the rectangle 
class: Draw. Erase, and Move. The Draw message causes 
invocation of a method for drawing a line from point to 
point where the points are represented by the instance 
variables Upperleft Corner. Upperright Corner. Lowerleft 

50 Corner, and Lowerright Comer. Similarly, an Erase message 
causes the lines to be erased Finally, a Move message 
causes the corner points to be translated or offset to another 
location. After a Move operation, a new Draw message 
might be sent, so that the rectangle appears on the screen to 

55 have moved from one position on the display to another. 
The concept of "notifications" in object oriented program- 
ming should also be kept in mind Like prograrnining of 
subroutines, the programming of objects will often result in 
the generation of return results ox messages that must be 

60 delivered back to an object that initiated an operation by 
sending a message. These return results or messages may 
also be called notifications. For example, a calculating 
object that is sent an "Add" message with instance variables 
**2" and "3" might return a notification of "5", which is the 

65 result of the addition operation 2+3. Similarly, a drawing 
object (Rectangle) that is sent a Draw message together with 
coordinates that are outside the range of the display screen 
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might return a notification (e.g. an error notification) that the 
operation cannot be completed as requested. Specific noti- 
fication messages utilized in the present invention are dis- 
cussed in greater detail below. 

Inheritance and Polymorphism 

Programmers utilizing an object-oriented programming 
environment can define a class in terms of an existing or 
higher up class. The existing class is called a superclass, and 
a new class may be called a subclass. A class without a 
superclass is said to be a root class. A subclass inherits all the 
instance variables and methods of an associated superclass. 
Subclasses can define additional instance variables and 
methods. Subclasses can also override methods defined by 
the superclass, so as to provide different operations in 
different contexts. 

Overriding a method means that the subclass responds to 
the same message as its associated superclass, but that the 
subclass object uses its own methods to respond to the 
message. 

For example, and looking at FIG. 13. suppose the pro- 
grammer wants to create a class to represent icons displayed 
on the display screen. This is shown as the Class: Icon 105. 
The instance variables might include a Bitmap (which is 
associated with a graphic image displayed on the screen), a 
Location (which is the x-y coordinates of the icon relative to 
the display screen), and an Underlying Object (which is 
another object that can be accessed or controlled by inter- 
acting with the icon, for example, an application program 
that is activated if the icon is clicked on). 

Exemplary messages that could be sent to a member (or 
instance) of the Icon class might include Select Activate. 
ShowMenu. Drag. Drop, Delete, and DoubleTap. The pro- 
grammer defines various steps of actions comprising com- 35 
puter program steps to correspond to each of these methods. 
For example, in response to a Select message, the program- 
mer may include program steps that causes the display of the 
Bitmap in reverse video on the display screen. Similarly, in 
response to a Activate message, the programmer might 40 
provide commands to send a Activate message to an Under- 
lying Object, and then display the Bitmap in reverse video 
and normal video in rapid succession to indicate visually 
that an activation process is underway. A ShowMenu mes- 
sage might cause the display of a context menu. A Drag 45 
message might cause translation or offset of the location of 
the Bitmap relative to the display screen. A Drop message 
might cause an action to take place that is appropri ate for a 
location at which an object is dropped, e.g. merely redraw- 
ing the icon at another screen location, or. if an object' s icon 50 
is dropped atop a printer icon, printing that object's data. A 
Delete message might cause deletion of the Bitmap from the 
display screen, as well as of the Underlying Object (which 
would result in freeing up memory). A DoubleTap message 
might cause execution of the first command on a command 55 
list associated with the icon. Other messages and methods 
will occur to those skilled in the art 

Still referring to FIG. 13, after a programmer has been 
provided with or created program code for an Icon class that 
carries out methods corresponding to messages, he or she 60 
can then use the Icon class to create subclasses. Consider for 
example the Icon subclasses in FIG. 13, Program Icon 108 
and Action Handle Icon 110. Both of these subclasses inherit 
all the instant variables from the Icon class. Each class, 
however, may define new instance variables and each sub- 65 
class can override certain methods. For example, the Pro- 
gram Icon 108 may include a Memory Allocation instance 
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variable that scores data corresponding to the desired 
memory utilized by the Underlying Object which is another 
computer program. 

Similarly, the Action Handle Icon subclass 110 will 
include additional Bitmap instance variables, e.g. Bitmapl. 
Bitmap2. Bitmap3. corresponding to different types of static 
action handles and drag icons that are displayed in response 
to various conditions in accordance with the invention. 

Each of these subclasses Program Icon 108 and Action 
Handle Icon 110 responds differently to an Activate Mes- 
sage. The Program Icon responds to an Activate Message by 
loading the associated program (the Underlying Object), 
while the Action Handle Icon responds to an Activate 
Message by popping up the context menu, as has been 
described elsewhere herein. 

As long as the objects are members of any of the asso- 
ciated Icon classes, they can be sent messages that are 
common to them all and the object will respond properly. 
The ability to send the same message to objects of different 
classes or subclasses is called ^lymorphisrcf. If the pro- 
grammer needs to create a different kind of Icon, all he or 
she needs to do Is define a subclass of Icon (by copying or 
duplicating the program code previously written for the Icon 
class of objects) and override such methods and provide 
additional instance variables as are required. 

With the foregoing background in mind, it will now be 
appreciated that the present invention is preferably made 
with object-oriented programming techniques. A complete 
discussion of all the possible objects that could be imple- 
mented in a computer program employing the teachings of 
the invention is beyond the scope of the invention and 
unnecessary for the skilled artisan. To enable the skilled 
artisan to carry out the invention, we will concentrate the 
remaining discussion on the various objects that will be 
constructed in accordance with the invention. 

FIG. 14 illustrates exemplary classes and relationships 
between classes that are provided in the present invention to 
implement action handles in accordance with the present 
invention. Discussion will not be provided of classes that are 
shown as the operating system object 120, the event handler 
object 125. a program application denominated HH APPLI- 
CATION 130. and a window object HHWINDOW 135. 
These and other objects are created by the programmer to 
write computer program code for a particular application. 
For purposes of the present invention, it will be understood 
that: 

an event handler object 125 receives user input such as a 
tap. click, or drag, and translates the user input into 
appropriate messages such as Tap. Click, or Drag, and 
provides these messages to the operating system 120. 

the operating system object 120 coordinates the resources 
of the computer system such as the event handler, disk 
drives, display screen, etc. It receives messages from 
various underlying objects, and passes messages back 
and forth between objects as appropriate. 

an HHAPPLICXTION object 130 is an object that imple- 
ments methods for a particular user purpose or 
application, for example, a Notes HHAPPLICATION 
object would handle storage, retrieval, and display of 
notes entered by a user. 

an HHWINDOW object 135 is an object that handles 
display of data in a window on a display screen. 
Intuitively, a ••window" object represented by a mem- 
ber of the HHWINDOW class corresponds to any type 
of window that can be displayed, in the abstract on the 
display screen. 
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TABLE m 



Event 


Location 


Action 


click/tap 


in action handle 


pop up context 






menu* send 


drag 


in fiction hwry^tr 


relocate data 




object and action 






handle to new 






display location at 






mouse up 


elks/tap 


in data object 


select data object 






for text editing or 
other action 






appropriate for 






particular type of 






data object 


drag 


in data object 


no effect, or 




object defined 






(e.g. handwriting) 


clkk/tap 


in window in 


activate window 




which action 


if beneath; 




handle and data 


otherwise nothing 




object appear 




drag 


in window io 


activate window 




which action 


if beneath; 




hsivflf and data 


otherwise nothing 




object appear 
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ao HHPANE object 138 is a subclass of HHWINDOW 
object that displays a selected subset of data on a 
display screen in a smaller k, pane*\ e.g. merely a range 
of selected text, or a user control button. A member of 
the subclass HHPANE can be a subwindow such as a 
dialog box. a button, or other species of a window that 
can be displayed on a display screen, 
a HHBUTTON object 140 is a subclass of HHPANE 
object that comprises a user control An action handle 
is an HHBUTTON type object 
a HHLISTBOX object 150 is a subclass of HHPANE 
object that comprises a displayed list of data items. A 
HHLISTBOX object has a pop-up or context menu that 
is displayed in response to a tap on an action handle. 
From the foregoing, it will be understood that action 
handles in accordance with the present invention are types of 
control buttons. HHBUTTON. and these are subclasses of 
the HHPANE class 138. The discussion which follows 
provides further information on the nature of the instance 
variables, messages, and methods required to implement 20 
objects that constitute action handles and associated data. 

TABLE m below illustrates various events that are 
handled by the event handler object 125 (FIG. 14). and the 
messages generated or actions taken. It will be understood 
that a "click/tap" event comprises a tap or a mouse down 23 
action, i.e.. the contacting of a stylus with the display screen 
or the depression of a mouse button. A "drag*' event in a 
pen-based system comprises a contacting of the stylus with 
the display screen at a first or starting location on the display 
screen, and movement of the stylus along a path on the 30 
display screen while maintaining the stylus in contact with 
the display screen, followed by lifting the stylus at a desired 
second or ending location. For a mouse or similar pointing 
device, a "drag" comprises a mouse down at a first or 
starting cursor location on the display screen, and movement 
of the cursor along a path on the display screen while 
maintaining the mouse button down, followed by releasing 
the mouse button at a desired second or ending location. 
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TABLES IV-VI below illustrate various aspects of the 
HHBUTTON class of objects 140 and HHLISTBOX class 
of objects 150 that are programmed In accordance with the 
teachings herein, fox implementing action handles in accor- 
dance with the present invention. Those skilled in the art will 
understand that the class definitions provided in the follow- 
ing tables, which should be self explanatory after the fore- 
going discussion on object-oriented programming, provide 
an enabling disclosure of the dass. instance variables, 
methods, and notifications unique to the HHBUTTON and 
HHLISTBOX classes, that differ from classes of a higher 
hierarchy such as HHPANE and HHWINDOW. for prepar- 
ing computer program code in accordance with the inven- 
tion. The tables describe particular display styles for asso- 
ciated graphic images associated with instances of the 
object, a general description of the class and its functions, 
messages provided to members of the class and correspond- 
ing methods carried out by members of the class, and 
notification messages provided in response to execution of 
program code to effectuate the methods described herein- 
above. 

TABLE IV 
HHBUTTON Class Definition 



Class HHBUTTON 
Instance Variables: N/A 
Messages & Notifications 
Styles: 

HHBS_J3RAGICONBMP 
HHBS_J)RAOICONBRD 



HHBS_DRAGICONBTN 
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Listed below 

Button is an action 
handle 

Button is an action 
handle with a bitmap 
with a raised border 
Button is an action 
hanril* bitmap with a 
border. 



Description This is a control object that designates a small 
rectangular "child" window that represents a button the user 
can activate by tapping or other user event There are 
different types of buttons for different uses, each considered 
a subclass, e.g. action handles or drag icons. 

Drag icons are used to implement a drag handle for data 
that does not have a drag handle implemented by a control. 
For example, list boxes, edit controls and note controls all 
support drag handles, but one could use a drag icon to 
associate a drag handle with data in a group of controls. 

Most push buttons resemble the raised-button look of a 
numeric keypad. They represent something the user can tap 
to perform an action or set a state. Regular push buttons are 
displayed in a depressed position when the user taps it and 
as in a raised position when released. In contrast a sticky 
button remains in the depressed position until the next click. 



Messages: 



Methods: 



It will be further understood that the click/tap and drag 
actions far the window in which an action handle and data 65 
object appear are the same actions as those taken in the 
known WINDOWS 111 graphic user interface. 



HHBM_GETBITMAP 
HHBM_SETBITMAP 
HUB M_QETD RAQICONCMD 

HHB M_0ETD RAGBCONCM 



Retrieves the handle to the 
bitmap displayed on tfac 
button 

Specifies die handle of the 
bitmap used to display the 



l ^ cntifi^ ft {he co^omajsd 
invoked when a user taps on 
the button; 

Q f n f ia tff the message 
WM_J)RAODROPREQUECT 
Retrieves a list of 



04/02/2004, EAST Version: 1.4.1 



5,760,773 



29 

-continued 



TABLE V 



KHUSTBOX Class TVfinirirm 



Class 

Instance Variables^ 
Messages & Notifications 
Description 



KHUSTBOX 
NVA 

Listed Below 

The HHListbox control b used 

tD list ^nfnnimtvm fyf 

applications, such Address, 
TcDo, and f^H Kh r 
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DLST 


mmmanrfe (HTMTI ST) tr> 




be displiycd in a pop op 




cooiexl menu 


HHBM.SETDRAGZCONCM 


Specifies the list of 


DLST 


commands (ffCMDL&f I) far 




(be actioo hffnHlf that 




appears when a user tapes 




the buttOD 



Notifications: The following describes the ootification 

messages that can be sen) by a HHBUTTON object 
to its parent via a WM_COMMAND message 
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Use box controls with the HHLS_DRAGABLE style 
support action handles. The application can set a default 
handheld icon, a handheld icon for a multiple selection and 
specific handheld icons for each list box item. If no handheld 
icon is set for an item, the default icon will be used. Though 
the list box will handle the drawing of the drag icon, the 
parent has the responsibility of responding to the 
WM_DRAGDROPREQUEST message in order to support 
drag and drop from a listbax. 

Action handles will be on the left of all lines in a list box 
item. 





C^nt qjh_» 1 1 t^Mh tans 
OCUI wllHl UK WT UI±B 




the action handle 


HHBN_J)BLCLK 


Indicates that the user 




double clicked or double 




tapped on an action 






HHBN_D RAGIC 0 NCMD 


Sent when the user taps 




on an action handle pop 




op menu item. The 




associated application 




program must then send 




the message 




HHBM^GETDRAGICONCMD 




to identify the 




iiu n*T^ invoked* 


HHBN_ v DRAGICONCMDLST 


Allows the application to 




set or change (he 




command list prior to 




showing it. This message 




is sent just before 




displaying the command 




list. 


HHBN_TDvffiOUT 


Smt to indicate a timr. 




out operation, if 




necessary 


WM_J>RAGDROPR£QUEST 


Notifies the parent object 




of action handle that a 




dragging opsjation is 




dftffirodv vpA to handle a 




drag and chop operation. 
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Styles The following describes the HHListbax control 
styles fm the HHLISTBOX class) specified in the 
dw Style parameter of the Create Window 
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function. 


Style 


Description 


HHLS__D RAO ABLE 


Specifies that the list box has 




action handles associated 




therewith. 


HHLS_^EXTENDEDS£L 


Allows a range of items to be 




sekcted 


HHLS_^NOTTFY 


Smrfc an input message to the 




parent window whenever the 




user taps or double-clicks a 







Messages The following descr i be s the messages that can be 
sent to an HHListBax control. Again, action 
handles are synonymous with drag icon. 
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The HHLISTBOX class of objects is the class that con- 50 
trols display of a pop-up or context menu of selectable items 
or commands. 

There are two types of selection modes depending on 
whether the HHLS _EXTENDEDSEL style is set single- 
selection mode and extended- selection mode. 55 

In single-selection mode, a pen down event on a list box 
item removes any previous selection and displays the item 
on which the pen is positioned as highlighted. The control 
tracks the pen until the pen is lifted and the new item is 
selected. That is, only a single item is ever selected at any 60 
one time. 

When the list box control Is in extended-selection mode, 
placing the pen down on an item removes any previous 
selection and highlights the item. When the pen is lifted, all 
items between the item on which the pen was placed and the 65 
item on which the pen was lifted become selected (not all 
items that have been touched). 



Message 


Method 


HHLB_GETDEFAULTDRAGICON 


Retrieves information 




about the default drag 




handle. 


HHLB_SETDEFAULTDRAOION 


Specifies the default 




drag icon attributes 


HHLB_GETD RACHC ONCMD 


Retrieves the identity 




of the command 




chosen from a 




command list* 


HHLB -GETDRAOICONCOL 


Retrieves the width. 




in pixels, of the drag 




icon column 


HHLB_SETORAOKONCOL 


Specifies the size, in 




pixels, of the left 




column where drag 




icons are displayed. 


HHLB_G&rrTKMDRA01CON 


Retrieves the 




information about the 




list box drag icon. 


HHLB_SETTTEMDRAGICON 


Specifies the drag 




icon mfbruiation of 




the list box item. 


Notifications The following describes die actions 


of the notification Me 


ssages sent by the HHListBox con- 


control. 




Notification Messages 


Action 


HHLBN_J)RAGICONCMDLST 


Sent when the Ittrtmx 




is about to display its 






HHLBN_J>RAGICONCMD 


Sent when the listhox 




has received the 




WM^COMMAND 




mcsM£t from the 




fX-'iLii n^ry^ list ftcxo* 


HHLB N_ERRSPACE 


Sent when list box 




cannot allocate 




enough memory to 




meet a specific 




request. 
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The following TABLE V contains a reference of exem- 
plary control messages that are employed in the preferred 
embodiment of the present invention. 

TABLE VI 

Control Messages 
HHBM_G£TBnMAP 



HHBM_CTTBTTMAP 
wParam = OL; /• not used, must bo zero */ 
IParam = (LPARAM) MAKELPARAM(ffresaed); /* 
pressed or normal ■/ 
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Parameters 


Ttria message has do parameters. 


Return Value 


Returns a T*«pH> id the consume) Hit, or 




NULL if oone is set ff the style is invalid, 




NULL is always returned. 


HHBH_SETBITMAP 



HHBM^SETBJTMAP 
wParam = (WPARAM) hBitmap; 

IParam = (LPARAM) MAKELPARAKKffressod); 
/* pressed or nonna] bitmap */ 



An application sends an HHBM_GETBITMAP message to 
retrieve the handle to the bitmap being displayed on the 
button. Applies only to buttons with the style HHBS 
PUSHBUTTON. HHBS_VE3mCALBUTTON. HHBS_ 
STICKYB UTTON . HHBS_AUTOSTICKYBUTTON. 
HHBS_DRAGICONBTN and HHBS _J)RAGIC0NBMP. 



Parameters Pressed 



Por HHBS_J>RAGKONBMP and 25 

HHBSJDRAGKONBTN styles, determines 

whether to retrieve the bitmap used for the 

pressed or non-pressed version of the 

button. Zero returns the ooD-pressod 

version. 

Return Value The return value is the handle to the bitmap 
if one exists for the button, else NULL The 
return value is always NULL for buttons 
that do not have the right style. For 
HHBS_J>RAGiCONBMP and 
HHBS _PRAGJCONBTN, if the low-order 
word of IParaxn is non-zero, it will 
retrieve the presses version of the bitmap. 
HHBM^SBTBITMAP does the same to set 
it 

HHBM_GETDRAGICONCMD 



HHBM_GETDRAGICONCMD 

wParam = 0; /*not used, must be zero •/ 40 

IParam = QL; /*not used, must be zero •/ 



This message identifies the command invoked when the user 43 
taps on a control with the button style HHBS_ 
DRAGICONBTN, HHBS __DRAGICONBMP, or HHBS_ 
DRAGICONBRD. An application sends an HHBM_ 
GETDRAGICONCMD message during the processing of 
the HHBN _J>RAGICONCMD notification to retrieve the M 
command id of the command list item picked by the user. 



Parameters This message has no parameters 

Return Value Returns the mrnrmrnrl ID, or zero. 

35 

HHBM^GETDRAQKXJNCMDLST 



HHBM-C&TTJRAGICONCMDLST 

wParam = 0; /*nol used, must be zero •/ 

IParam = CL; f+vot used, must be zero "/ 



An application sends HHBM_SETDRAGICONCMDLST 
to a button with style HHBS_DRAGICONBTN, HHBS_ 
DRAGICONBMP. or HHBS_DRAGICONBRD to retrieve 65 
the handle of the command list (ie., context or pop up menu) 
it will show when the user taps or dicks on it 



An application sends a HHBM__SETBITMAP message to 
set the handle of the bitmap to be used when displaying the 
action handle. For color bitmaps, the blue bits will appear 
transparent Color bitmaps are device dependent 

The bitmap will be centered if there is no text else it will 
appear to the left of the text. This applies only to buttons 
with the styles HHBS_VBRnCA13 UTTON. 



KHBS_STBCKYBUTTON, HHBS_AUTOSTTCKYB UTTON, 
HHBS_J3RAGICONBTN, or HHBS_D RAGICONB MP. 



Parameters hBitznap 

Value of wParam. 

Handle to the bitmap to be act 

(Pressed 

For HHBS_DRAGKONBTN, or 
HHBS_D RAGICONBMP, and 
HHBS_DRAGlCONBIU) styles, 
determines whether to set the bitmap 
used fox the pressed or non-pressed 
version of the button. Zero sets the 
noo-pressed version. 
Return Value The return value is TRUE for a 

successful setting, else FALSE. 



HHBM SETDRAOIC ONCMDLST 



HHBH_SETDRAOICONCMDLST 
wParam = hcmdlst; 

IParam = QL; /*not used, must be zero */ 



An application sends HHBM_SETDRAGICONCM DLST 
to a button with style HHBS_PRAGICONBTN or HHBS_ 
DRAGICONBMP to set the handle of the command list it 
will show when the user taps or clicks on it. 



Parameters 


hcmdlst 




Value of wParam. Specifies the 






Return Value 


Returns a nonzero if set, or a 0 for 




an invalid style. 



HHEM—CHECKDRAGICON 



HHBM_CHECKDRAGICON 

wParam = O, /*ool used, must be zero */ 

IParam = OL; /•not used, must be zero •/ 



An application sends an HHEM_CHECKDRAGlCON 
message to cause the control to update its drag icon if 
needed. 
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-continued 



Parameters 
Return Value 



Tfats message has do 
None 



HHLB_GETDEFAULTDRAGICON 



HHBM_GETD£FAUUD RAGICON 

wParam = 0; /*not used must be zero */ 

IParam = (LPARAM)lptbdragicon; 

/*ptr. to Ibdragcon structure */ 
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Ad application sends the HHLB_ 
GETDEFAULTDRAGICON message to retrieve the drag 
icon information for the default drag icons for the list box. 
Sending this message causes the Ibdragicon structure speci- 
fied by the IParam to be filled 



is 



Parameters Iplbdragtcco 

Specifies a long pointer to a 
LB D RAGICON structure, to 
fill with default drag icon 

typedef struct tagLBD RAGICON 
i 

HBITMAP hbmUp; 
HBITMAP hbmDown; 
LBDRAGKON; 
} 

Return Value Returns HHLB_OKAY if the structure was 
filled is with the defauh drag icon 
information. It returns HHLB^ERR if an 
error occurs or if the listbox is without the 
styfe HHLS_0RAGABLE . 
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HHLB_GETD RAOIC ONCMD 



HHBM_GETT>RAGICONC MD 

wParam = 0. /*not used, must be zero */ 

IPurazn = OL; /*not used, must be zero */ 



IParam = 01PARAM)lribaragicon; /•pot. to Ibdrtgicai 
structure •/ 



An application sends the HHLB_Gb I I 1 bMDRAGlCON 
message to retrieve the handheld icon information of the 
listbox item in a Ustbox with the HHLS_DRAGGABLE 
style 

Parameters odea 

Specifies the zero-based index of the 
item's handle to be retrieved. 
fptbdra£)C08 

Specifics a long pointer to a 
LBD RAGICON structure to be filled, 
typedef struct tagLB D RAGICON 
( 

HBITMAP hfacnUp; 
HBITMAP hbmDowu: 
LBDRAGKON; 
> 

Return Value Returns the height, in pixels, of a listbox 

it e m 

HHLB_SETDEFAULTD RAGICON 

HHBM__SETDEFAULTDRAGlCON 
wParam = 0; /*not used, must be zero •/ 

IParam = (LPARAM)];plbdraxiccm 

/•ptr. lo Ibdragicon structure */ 



An application sends the HHLB_ 
SETDEFAULTDRAGICON message to set the default drag 
icons used by the list box to the bitmaps in the specified 
Ibdragicon structure. 
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An application sends an HHLB_GETDRAGICONCMD 
message to retrieve the command id of the item selected by 
the user from the command list This message is only valid 
during processing of the HHLBN_DRAGICONCMD noti- 
fication 



Parameters 


This m^egflg* has no parameters. 


Return Value 


Returns the ID of the command list item, or 




HHLB_BRR in a listbox without the style 




HHLS__DRAGAHLE. 


HHLB_GETDRAGICONCOL 



HHBH.GBTDRAGICONCOL 

wParam = O. /•not used, must be zero •/ 

tParam = QL; S*oot used, must be zero */ 
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Parameters Iptbetragtcon 

Long pointer to a LBDRAGICON 
structure or NULL for the item to 
use the default icon. 

typedef struct tagLBDRAGICON 

{ 

HBITMAP hbmUp; 
LBDRAGICON; 

) 

Return Value Returns HHLB.OKAY if default drag u 
15 set. or else HHLB_ERR if an error 
occurred m a listbox without the style 
HHLS_J>RAOABLE. 



hhlb_setdragk;oncol 



HHLB_SETDRAGICONCOL 
wParam = (WPARAM)colwidth; f* new width of 
drag icon cohnnD */ 
= OL; /* not used, must be zero */ 



An application sends an HHLB_GETDRAGICONCX)L 
message to retrieve the size, in pixels, of the column that 55 
displays icons on the left side of a listbox. 



Parameters This message has no parameters. 

Return Value Return* the width of the column on the left 
side of the list box to show drag icons in a 
listbox with me style HKLS_JD RAG ABLE, 
or else HHLB_ERR. 



HHLB_GBTTmMDRAGICON 



HHBM_<iE 1 1 1 HMD RAGICON 
wParam = index; /*not used to item •/ 



An application sends an HHLB_SETDRAGICONCOL 
message to set the size, in pixels, of the column that displays 
drag icons on the left side of a listbox. 



Parameters 


colwidm 




Value of wparanu The wkhh, in 




pixels, of the new width of the drag 




icon column. 


Return Value 


Returns the width of the column on 




the left side of the listbox to show 




drag icons b a listbox won the style 




HHLS_DRAGABLE, or else 




HHLB_ERR. 
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HHLB_SFITTEMDRAQICON 



HHLB_SETTrEDRAGICON 

wParam = index; /* index n> item •/ 

IPttram = (LPARAM)^Lbdrapoon; 

/* ptr to Ibdrijpcan struct */ 



An application sends the HHLB_SKIII'KMDRAGICON 
message to set the handheld icon information of a list box 
item's icon. The list box must have the style HHLS_ 
DRAGGABLE set 



index 

Specifies the zoo-based index of the 
item 

typedef struct tagLBDRAGICON 
{ 

HBITMAP hbmUp; 
HBITMAP hbmDown; 
LBDRAGICON; 
} 



Return Value 



The return value is 0 or HHLB_£RJt 
if an error occurred 



Parameters wparain 

Specifies the ■ < ^T ,fi fW of the burton 
IParam 

5 Specifies the handle of the control box in 

the taw-order word, and specifies the 
HHB N_D RAGICONC MDLST notification 
wmtgr m the high-order word. 
HHLBN_DRAGJCONCMD 

10 

This notification message is sent by a listbox with the style 
HHLS_DRAGABLE. It indicates that the user has picked 
an item from the listbox* s command list The application can 
query the command identity of the selected command list 
15 item by sending an HHLB_GETDRAGICONCMD. 



Parameters wParam 

Specifies the iffrntifVr of the burton 
IParam 

Specifies the handle of (he list box in the 
low-order ward, and specifies the 
HHBN_J>RACHCONCMD Dotifk&tian 
message in the high-order word. 
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Notification Messages 

This section contains a reference of notification messages 
that are employed in the preferred embodiment of the 
present invention. 

HHBNJDRAGICONCMD 

The HHBNJDRAGICONCMD notification message is 
sent by buttons with the button style HHBS_ 
DRAGICONBTN or HHBS_DRAGICONBMP. It indi- 
cates that the control has just received a WM_COMMAND 
from its command list indicating that the user has just picked 
a command list item. The application can query which 
command item was picked by sending the HHBM_ 
GBTDRAGICONCMD message. 



Parameters index 

Specifies the zoo-based index of the 
item 

Iplbdragioon 

tons pointer to a LBDRAGKON 
structure or NULL for the item to 
use the default icon. 

typedef struct tagLBDRAGICON 

I 

HBITMAP hbmUp; 
HBTTMAP bhmDown; 
} LBDRAGICON; 
Return Value The return value is 0 or HHLB__J£RR if s 

error occurred. 



HHLBN_DRAGICONCMDLST 

This notification message is sent by a listbox with the 
style HHLS_DRAGABLE. It indicates mat the listbox is 
about to display its command list The application can find 
3d out what listbox items are associated with the active drag 
icon by sending an HHLB_GETDRAG1C0NTTEMS mes- 
sage to the listbox. If necessary, modify the command list 
associated with the list box by using the HHLB_ 
GETCMDLST and HHLB_SETCMDLST messages. 
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HHBN_DRAGICONCMDLST 

The HHBN_DRAGICONCMDLST notification mes- 
sage is sent to the application to allow it to set or change the 
command list prior to showing it This message is sent by a 
control with button style HHBS_DRAGICONBTN or 
HHBS_DRAGICONBMP. 



Parameters wparam 

Specifies the identifier of the button 
rParam 

Specifics the handle of the list box in the 
lew-order word, and specifies (he 
HHBN _DRAGICONC MDLST notification 
message in the high-order word. 



It will now be understood that the present invention has 
45 been described in relation to particular embodiments which 
are intended in all respects to be illustrative rather than 
restrictive. Alternative embodiments will become apparent 
to those skilled in the art to which the present invention 
pertains without departing from its spirit and scope. 
50 Accordingly, the scope of the present invention is defined by 
the appended claims rather than by the foregoing descrip- 
tion. 

What is claimed is: 

1. In a computer controlled display system having a 
53 display coupled to a central processing unit (CPU) for 
displaying images, a method for displaying and controlling 
actions relating to a data object displayed on the display, 
comprising the steps of: 

generating a graphic image associated with the data object 

on the display; 
generating a graphic image of an action handle proximate 

to the data object; 
providing an activatable region on the display associated 

with the action handle graphic image; 
in response to the user's positioning of a pointer on the 
display with a pointing control device operatively asso- 
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ciated with the CPU. providing a signal to the CPU 
upon correspondence of the pointer with the activatable 
region of the action handle; 
in response to a signal indicative of a first type of 
interaction between the pointer and the activatable 5 
region associated with the action handle, displaying at 
least one selectable command relating to the data object 
in a context menu including the at least one selectable 
command; and 

in response to signal indicative of a second type of 10 
interaction between the pointer and the activatable 
region associated with the action handle, moving the 
graphic image of the data object on the display. 

2. The method of claim 1. wherein the first type of 
interaction is a tap. 15 

3. The method of claim 1, wherein the second type of 
interaction is a drag. 

4. The method of claim 1. wherein the step of displaying 
at least one selectable command comprises: 

displaying a plurality of selectable commands in the 
context menu. 

5. The method of claim 1. further comprising the steps of: 
providing an activatable second region in the context 

menu on the display associated with the selectable 25 
command; and 
in response to interaction between the second region and 
the pointer, executing a computer command corre- 
sponding to the selectable command. 

6. The method of claim 5. wherein the interaction between 30 
the pointer and the second region comprises a tap. 

7. The method of claim 1. wherein the step of moving the 
graphic image of the data object on the display comprises a 
drag-and-drop operation. 

8. The method of claim 1. wherein the step of moving the 35 
graphic image of the data object on the display comprises: 

moving the graphic image of the action handle on the 
display during a dragging operation from a first posi- 
tion to a second position on the display; 

in response to a release command provided at the second 40 
position oo the display, moving the graphic image of 
the data object to a new position on the display asso- 
ciated with the second position. 

9. The method of claim 1. wherein the data object is a data 
item associated with a predetermined application program. 45 

10. The method of claim 9, wherein the data item is a 
selected string of text associated with a text processing 
application program. 

11. The method of claim L wherein the display system 
includes a direct pointing device for providing signals to the 50 
CPU. 

IX A computer controlled display system, comprising: 
a central processing unit (CPU); 

a memory coupled to said CPU for storing programs and ^ 
data; 

a graphic display coupled to said CPU for displaying 
information; 

a pointing device coupled to said CPU for allowing an 
operator to point to items displayed on said display and $0 
for providing signals to said CPU indicative of prede- 
termined actions; 
said CPU being operative to: 
store data corresponding to a data object in said 

memory; 65 
display a data object graphic image associated with said 
data object on said display; 
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display an action handle proximate to said data object 
graphic image on said display, said action handle 
comprising a predetermined activatable region on 
said display associated with said action handle; 
in response to a signal from said pointing device 
indicative of a first type of interaction between said 
pointing device and said activatable region of said 
action handle, display at least one selectable com- 
mand operarJvely associated with said data object in 
a context menu including the at least one selectable 
command; and 
in response to a signal from said pointing device 
indicative of a second type of interaction between 
said pointing device and said activatable region of 
said action handle, reposition said data object 
graphic image on said display. 

13. The system of claim 12, wherein said first type of 
interaction between said pointing device and said activatable 
region comprises a tap. 

14. The system of claim 12. wherein said second type of 
interaction is a drag. 

15. The system of claim 12, wherein the 
context menu on the display includes a plurality of 

selectable commands. 

16. The system of claim 1Z wherein said CPU is further 
operative for: 

providing an activatable second region on the display 

associated with said selectable command; and 
in response to interaction between the second region and 
said pointing device, executing instructions corre- 
sponding to said selectable command. 

17. The system of claim 16. wherein said interaction 
between said pointing device and said activatable second 
region comprises a tap. 

18. The system of claim 12. wherein said CPU operation 
of repositioning said data object graphic image on said 
display comprises a drag-and-drop operation. 

19. The system of claim 12. wherein said CPU operation 
of repositioning said data object graphic image on said 
display comprises: 

moving said graphic image of said action handle on said 
display during a dragging operation from a first posi- 
tion to a second position on said display; 
in response to a release command provided at said second 
position, moving said graphic image of said data object 
to a new position on said display associated with said 
second position. 

20. The system of claim 12. wherein said data object is a 
data item associated with a predetermined application pro- 
gram. 

21. The method of claim 20. wherein said data item is a 
selected string of text associated with a text processing 
application program. 

22. The method of claim 20. wherein said data item 
comprises a calendar entry associated with a calendar appli- 
cation program. 

23. In a computer controlled display system having a 
display coupled to a central processing unit (CPU) for 
displaying images, a method for displaying and controlling 
actions relating to a selected text string displayed on the 
display, comprising the steps of: 

generating a graphic image of a string of text on the 
display; 

in response to selection of predetermined selected text 
within the string of text generating a graphic image of 
an action handle in a region proximate to the predeter- 
mined selected text; 
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providing an activatable region on the display associated 
with the action handle; 

in response to a user's positioning of a pointer on the 
display to Che activatable region associated with the 
action handle using a pointing control device opera- 5 
tively associated with the CPU. providing signals to the 
CPU indicative of correspondence of the pointer with 
the action handle; 

in response to a signal indicative of a first type of 
interaction between the pointer and the activatable 10 
region associated with the action handle, displaying at 
least one selectable command relating to the predeter- 
mined selected text in a context menu including the at 
least one selectable command; and 

in response to a signal indicative of a second type of 
interaction between the pointer and the activatable 
region associated with the action handle, moving the 
predetermined selected text on the display. 

24. The method of claim 23, wherein the step of display- M 
ing at least one selectable command comprises: 

displaying a plurality of selectable commands in the 
context menu. 

25. The method of claim 24. wherein the context menu 
comprises a list of a redetermined plurality of selectable 25 
commands, and wherein the selectable commands at the 
beginning of the list are of a first application specific type 
and wherein the commands at the end of the list arc of a 
second, operating system specific type. 

26. The method of claim 23. further comprising the steps 
of: 



providing an activatable second region on the display 
associated with the selectable command; and 

in response to interaction between the second region and 
the pointer, executing a computer command corre- 
sponding to a particular selectable command. 

27. The method of claim 26, wherein the interaction 
between the pointer and the second region comprises a tap. 

28. The method of claim 23. wherein the step of moving 
the predetermined selected text on the display comprises a 
drag-and-drop operation. 

29. The method of claim 23. wherein the step of moving 
the pedetennined selected text on the display comprises: 

moving the action handle on the display during a dragging 
operation from a first position to a second position on 
the display; 

in response to a release command provided at the second 
position on the display, moving the predetermined 
selected text to a new position on the display associated 
with the second position. 

30. The method of claim 23, wherein the string of text is 
displayed on the display in a scrollable window, and further 
comprising the steps of 

in response to a signal indicative of the second type of 
interaction between the pointer and the action handle 
and movement of the action handle to a boundary of the 
scrollable window, scrolling the text in the scrollable 
window; and 

continuing to display the action handle pinned to the 
boundary of the window during scrolling of the text 
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